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Disclaimer 

Administratively,  this  study  was  funded  by  the  Army  Research  Lab  as  part  of  a  year-long 
effort  in  support  of  the  Statement  of  Work  entitled,  “EASE  Stakeholder  Analysis.”  The 
U.S.  Government  is  authorized  to  reproduce  and  distribute  reprints  for  governmental 
purposes  notwithstanding  any  copyright  annotation  thereon.  The  views  and  conclusions 
contained  herein  are  those  of  the  authors  and  should  not  be  interpreted  as  representing  the 
official  policies  or  endorsements,  either  expressed  or  implied,  of  the  ARL,  USMA,  or  the 
U.S.  Government. 


Executive  Summary 


Modeling  and  Simulation  (M&S)  plays  a  major  role  in  the  mission  performance  of  the  Department  of 
Defense  (DOD).  DOD  spends  more  than  $3  billion  per  year  on  M&S  to  support  acquisition,  training, 
experimentation,  planning,  testing  and  analysis.  M&S  uses  are  varied  and  include:  analysis  of  cost- 
mission  trades  for  new  weapon  systems,  developmental  or  operational  tests  of  new  systems,  analysis  of 
force  structure,  training,  and  effectiveness  analysis  of  weapon  systems.  Each  M&S  organization  has 
unique  missions,  purpose,  and  requires  varying  levels  of  analytical  fidelity  as  their  work  supports 
different  categories  of  decisions  (acquisition,  design,  training,  etc.).  This  decentralized  procurement, 
development,  and  use  of  combat  simulation  models  and  tools  presents  many  challenges. 

The  research  team  at  STTC  has  attempted  to  holistically  address  many  of  these  challenges  via  a  web- 
based  tool;  Executable  Architecture  Systems  Engineering  (EASE).  EASE  is  a  Systems  Engineering  tool 
that  allows  development  and  management  of  distributed  simulation  models  throughout  the  M&S  life 
cycle  from  identification  of  event  objectives  through  cloud-based  deployment.  As  a  web-based 
application,  EASE  provides  an  easy  to  use  interface  to  allow  M&S  users  to  more  easily  configure  and 
execute  M&S  on  a  cloud-based  set  of  computing  resources.  EASE  allows  M&S  users  to  customize 
execution  of  a  simulation  event  based  on  an  interview  process  that  identifies  system-wide  functional  and 
technical  requirements  and  then  determines  which  applications  and  hardware  allocations  are  necessary  for 
execution  to  achieve  these  functional  and  technical  requirements.  EASE  automatically  configures  the 
network  and  necessary  supporting  software  in  order  to  execute  the  applications  on  virtual  machines  using 
a  Platform  as  a  Service  architecture. 

The  EASE  development  team  commissioned  this  study  in  order  to  focus  planned  improvements 
to  EASE,  based  on  a  comprehensive  study  of  the  needs  and  preferences  of  potential  users  and 
other  stakeholders  to  determine  the  most  important  functions  and  attributes  for  the  product. 
Specifically,  conduct  a  detailed  stakeholder  analysis,  looking  very  broadly  at  the  various 
stakeholders  and  the  desired  functions  of  EASE,  in  order  to  devise  and  prioritize  possible 
additions  or  improvements  for  the  development  team  to  include  in  future  versions. 

We  utilize  both  the  Systems  Decision  Process  (SDP)  and  Value-Focused  Thinking  (VFT)  to  gather  and 
analyze  stakeholder  feedback.  User  feedback  is  clustered  and  organized  into  Findings,  Conclusions,  and 
Recommendations  (FCR)  to  highlight  trends,  capability  gaps,  and  major  issues.  The  FCR  tables  and 
stakeholder  feedback  are  then  used  as  the  foundation  of  a  Strengths,  Weaknesses,  Opportunities,  and 
Threats  (SWOT)  analysis.  Finally,  the  SWOT  analysis  and  stakeholder  feedback  are  translated  into  an 
EASE  future  development  strategy;  a  series  of  recommendations  regarding:  stakeholder  solution  space 
focus,  specific  M&S  organizations  with  interest,  prioritized  EASE  improvements,  prioritized  list  of  EASE 
enhancements,  and  potential  use  cases. 

EASE  Strengths.  Most  notably,  stakeholders  were  not  aware  of  another  “EASE-like”  product 
in  use  or  under  development.  They  value  EASE  capability  to  document  and  archive  model 
architecture  and  interoperability  requirements.  Stakeholders  saw  the  ability  to  maintain  and 
reuse  previous  combat  simulation  scenarios  and  runs  as  a  clear  strength.  The  surrogate 


capability  provided  in  EASE  was  highlighted  as  unique  and  positive.  Lastly,  EASE  has  potential 
to  both  reduce  their  hardware  and  software  footprint  as  well  as  provide  a  back-up  capability. 

EASE  Weaknesses.  Stakeholders  were  concerned  about  having  a  lack  of  in-house,  EASE 
expertise;  potentially  creating  a  single  point  of  failure.  They  note  that  they  lack  the  manpower, 
expertise,  and  experience  required  to  build  SDDs  that  would  properly  function  within  EASE. 
Because  only  a  few  models,  scenarios,  and  supporting  SDDs  are  currently  represented  in  EASE, 
stakeholders  feel  that  EASE  would  not  provide  any  additional  advantage  over  the  current  way  of 
doing  M&S  business  and  would  make  it  difficult  to”  sell”  to  their  managers  and  fellow  M&S 
users.  Lastly,  the  perceived  risk  associated  with  not  being  a  Program  of  Record  (POR)  was 
highlighted  as  a  major  weakness. 
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Figure  A.  EASE  SWOT  Analysis 


Recommended  EASE  Enhancements.  A  common  stakeholder  recommended  high  priority 
enhancement  is  integration  of  EASE  with  current  Mission  Command  and  C2  Systems. 
Stakeholders  also  recommended  adding  many  more  scenarios,  models,  and  supporting  SDDs  for 
the  most  commonly  used  combat  simulation  models  and  tools.  An  EASE  linkage  to  Force 
Builder  was  mentioned  as  a  medium  priority  recommended  enhancement.  Low  priority 


recommended  enhancements  include  a  robust  report  and  analysis  capability  and  linkage  of 
terrain  to  the  application  line-up. 

A  recommended  Stakeholder-Organization-Capability-M&S  Phase- Application  focus  for  future  EASE 
developments  and  enhancements  are  highlighted  in  Figure  B  below  (Priorities  highlighted  in  Green).  In 
general,  prioritized  EASE  improvements  should  focus  on  scenario  development/modification  capability, 
increasing  the  ease  of  integrating  disparate  models,  establishing  linkages  to  authoritative  data  sources,  and 
continuing  to  populate  the  application  database  with  accompanying  SDDs.  The  EASE  development  team 
can  assist  the  M&S  community  the  most  by  focusing  improvements  and  enhancements  on  efforts  that 
provide  value  to  the  Systems  Engineering  and  Development  phases  of  the  M&S  lifecycle.  We 
recommend  adding  VBS2,  Night  Vision  Tool  Kit,  and  JCATS  to  the  application  database  and  line-up 
with  appropriate  supporting  SDDs. 

A  final  general  recommendation  is  to  identify  an  appropriate  VV&A  authority,  discuss  the  specific 
VV&A  requirements  as  they  pertain  to  EASE,  and  begin  action  on  those  VV&A  related  tasks  that  can  be 
completed  now.  Highest  priority  should  be  given  to  EASE  enhancements  that  are  highlighted  in  green. 
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Figure  B.  Recommended  EASE  Enhancement  Focus 


We  recommend  two  use  cases  to  demonstrate  the  valuable  and  innovative  capabilities  of  EASE: 

MSCoE.  Utilize  EASE  to  support  their  upcoming  SIMEX.  EASE  can  improve  MSBL 
execution  of  simulation  both  in  the  short  term  as  well  as  the  long  term.  In  the  short  term,  EASE 
could  facilitate  the  automation  of  execution  of  M&S  across  their  lab  assets.  EASE  would  capture 
the  technical  complexity  of  their  simulation  environment  and  provide  a  simple  interface  to 
execute  M&S  as  well  gather  AAR  products  through  a  single  web  interface.  In  the  long  term, 
EASE  could  be  used  to  link  simulation  capabilities  with  low  level  technical  design  details.  This 
will  ultimately  lead  to  better  reuse  and  interoperability  providing  cheaper  and  more  accurate 
MSBL  M&S  usage. 


USMA.  Utilize  EASE  to  facilitate  DSE  work  in  support  of  their  Squad  X  and  Deployable 
Force  Protection  (DFP)  projects.  Specifically,  use  EASE  to  help  develop  system  of  system 
federations  that  support  each  program.  Key  capabilities  required  will  be  systems  engineering 
analysis,  federation  management  and  start/stop,  and  data  collection.  DSE  would  like  to  assess 
EASE  ability  to  build  command  and  control  data  models  and  simulation  federates  that  pass 
federation  data  to  command  and  control  systems  used  for  both  DFP  and  Squad  X.  Additionally, 
once  loaded  in  DSE  labs,  EASE  could  be  used  to  support  the  combat  simulation  and  architecture 
courses. 


Lastly,  a  value  hierarchy  and  proposed  set  of  metric  to  calculate  the  value  of  EASE  in  Return  on 
Investment  (ROI)  discussions  was  developed  and  highlighted  in  Figure  C  below. 


Figure  C.  EASE  Value  Hierarchy  and  Metrics 
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1.0  Introduction 


Modeling  and  Simulation  (M&S)  plays  a  major  role  in  the  mission  performance  of  the 
Department  of  Defense  (DOD).  DOD  spends  more  than  $3  billion  per  year  on  M&S  to  support 
acquisition,  training,  experimentation,  planning,  testing  and  analysis.1  M&S  uses  are  varied  and 
include:  analysis  of  cost-mission  trades  for  new  weapon  systems,  developmental  or  operational 
tests  of  new  systems,  analysis  of  force  structure,  training,  and  effectiveness  analysis  of  weapon 
systems.  The  strategic  vision  for  DOD  modeling  and  simulation  is  to  “empower  DOD  with 
Modeling  and  Simulation  capabilities  that  effectively  and  efficiently  support  the  full  spectrum  of 
the  department’s  activities  and  operations.”2 

1.1  Background 

Each  M&S  organization  has  unique  missions,  purpose,  and  requires  varying  levels  of  analytical 
fidelity  as  their  work  supports  different  categories  of  decisions  (acquisition,  design,  training, 
etc.).  Because  of  these  unique  missions  and  analytical  requirements,  each  laboratory  or 
analytical  organization  utilizes  their  own  collection  of  M&S  tools  and  in-house  scripts. 
Additionally,  there  is  not  a  one  size  fits  all  combat  simulation  software  package.  Rather,  there  are 
approximately  600  combat  simulation  models  and  tools  all  developed  independently  for  a  unique 
analytical  purpose.  This  decentralized  procurement,  development,  and  use  of  combat  simulation 
models  and  tools  presents  many  challenges. 

Model  and  tool  documentation  is  typically  non-existent  or  lacking  which  prevents  an 
organization  from  using  another’s  models  or  tools.  Organizations  have  created  in-house  add-ons 
required  to  pre  and  post  process  data  and  information  for  their  specific  needs.  No  central 
repository  exists  where  a  lab  could  possibly  leverage  (check  out)  one  of  these  600  M&S  tools. 
Version  management  and  control  is  a  persistent  problem.  A  new  version  of  a  model,  tool,  or 
operating  system  introduced  into  a  lab  or  analytical  agency  creates  new  configuration  issues  and 
runtime  errors.  There  is  tremendous  redundancy  in  combat  functions  represented  within  models. 
For  example,  direct  and  indirect  fire  effects  are  incorporated  into  almost  every  combat  simulation 
model.  A  tremendous  amount  of  resources  including  hardware,  software,  laboratory  space,  and 
personnel  are  required  to  prepare  and  maintain  appropriate  hardware  and  software  and  these 
resources  are  duplicated  from  lab  to  lab.  Instead  of  looking,  unsuccessfully,  for  a  combat 
simulation  model  that  meets  all  of  their  unique  requirements,  most  labs  and  analysis  agencies  are 
creating  federations  of  numerous  model  components. 

Research  efforts  over  the  last  several  years  have  focused  on  addressing  many  of  these  challenges, 
individually,  but  not  as  a  whole.  DOD  has  encouraged  migration  of  software  to  the  “cloud”  and 
use  of  virtual  machines  (VM)  to  reduce  the  Enterprise  hardware  and  software  footprint. 

1  Shaffer,  The  Value  of  Modeling  and  Simulation  for  the  Department  of  Defense.  M&S  Journal,  Fall  2012,  p.2 

2  Office  of  the  Director  of  Defense  research  and  Engineering.  2007.  Retrieved  from  http://www.msco.mil 
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Numerous  research  organizations  have  had  varying  degrees  of  success  with  federation  of  models 
and  supporting  architecture. 

Modeling  Architecture  for  Technology  Research  and  Experimentation  (MATREX),  a 
composable  Modeling  &  Simulation  (M&S)  environment,  appears  the  most  successful  recent 
attempt.  The  Modeling  and  Simulation  Coordination  Office  (MSCO)  has  attempted  to  catalog 
and  document  the  myriad  of  models  and  tools  in  the  M&S  community.  The  research  team  at 
STTC  has  attempted  to  holistically  address  many  of  these  challenges  via  a  web-based  tool; 
Executable  Architecture  Systems  Engineering  (EASE),  which  facilitates  development  and 
management  of  distributed  simulation  models  throughout  the  M&S  life  cycle. 


1.2  What  is  EASE 

Executable  Architecture  Systems  Engineering  (EASE)  is  a  Systems  Engineering  tool  that  allows 
development  and  management  of  distributed  simulation  models  throughout  the  M&S  life  cycle 
from  identification  of  event  objectives  through  cloud-based  deployment.  As  a  web-based 
application,  EASE  provides  an  easy  to  use  interface  to  allow  M&S  users  to  more  easily 
configure  and  execute  M&S  on  a  cloud-based  set  of  computing  resources.  EASE  allows  M&S 
users  to  customize  execution  of  a  simulation  event  based  on  an  interview  process  that  identifies 
system-wide  functional  and  technical  requirements  and  then  determines  which  applications  and 
hardware  allocations  are  necessary  for  execution  to  achieve  these  functional  and  technical 
requirements.  EASE  automatically  configures  the  network  and  necessary  supporting  software  in 
order  to  execute  the  applications  on  virtual  machines  using  a  Platform  as  a  Service  architecture. 

As  highlighted  in  Figure  1  below,  EASE  provides  multiple,  integrated  interfaces  for  users, 
integrators,  developers,  and  system  engineers  to  accomplish  the  typical  M&S  tasks  associated 
with  their  user-category. 

#9 

44 
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The  major  components  of  EASE  are  the  Software  Design  Description  (SDD),  EASE  Interview, 
Deployment  Management  System,  and  EASE  Coordinator.  Figure  2  below  illustrates  the 
relationship  between  these  major  components  and  the  user-categories. 
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Figure  1.  EASE  User-level  Interfaces 
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The  SDD  captures  the  systems  engineering  information  on  the  available  simulation  applications, 
their  capabilities  and  how  they  interoperate  in  a  simulation  environment.  The  SDD  also  allows 
the  system  engineering  user  to  add  new  simulation  applications. 

The  EASE  Interview  System  allows  the  user  to  traverse  captured  system  engineering  information 
to  select  and  compose  a  simulation  system.  The  user  is  presented  with  a  list  of  options  based 
upon  scenario  criteria  and  functional  capabilities  and  has  the  ability  to  customize  components  of 
the  scenario.  Additional  advanced  capabilities  allow  the  user  to  inject  custom  properties  and 
create  surrogates  to  fill  in  specific  capabilities. 

Once  the  scenario  has  been  designed  and  the  components  chosen,  The  EASE  Deployment 
Management  System  determines  the  necessary  assets  for  execution  and  deploys  software  and 
configuration  files.  It  employs  Platform  as  a  Service  (PaaS)  to  utilize  virtual  and  hardware  assets 
in  support  of  a  simulation  exercise.  Its  tasking  service  then  determines  how  and  when  to  run  a 
simulation  execution. 

Finally,  the  EASE  Coordinator  is  responsible  for  the  actual  execution  of  the  simulation  exercise. 
The  Coordinator  handles  the  Time  Sequence  of  Events  provided  by  the  tasking  system  and 
controls  the  launch,  initialization,  shutdown  and  cleanup  of  each  process.  The  Coordinator  is 
also  responsible  for  progression  of  the  overall  simulation  execution  ensuring  all  processes 
perform  the  necessary  tasks  at  the  proper  time. 


Systems  Engineer 


Developers  and 
Integration  Engineers 


Hardware  and 
Network  Engineers 


System  Design 
Description 

CD 

CD 

JO 

co 

EASE  Interview 

Deploy  Asset 
Management 

Workflow  System 

(SDD) 

LD 

CO 

Software  1  Hardware 

Figure  2.  EASE  Major  Components 


The  fundamental  goals  of  EASE  are  to  manage  the  requirements  and  design  process,  maximize 
the  reuse  of  models  and  streamline  software  and  hardware  management  during  all  M&S 
development  phases. 

1.3  Problem  Statement. 

The  EASE  development  team  commissioned  this  study  in  order  to  focus  planned  improvements 
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to  EASE,  based  on  a  comprehensive  study  of  the  needs  and  preferences  of  potential  users  and 
other  stakeholders  to  determine  the  most  important  functions  and  attributes  for  the  product. 
Specifically,  conduct  a  detailed  stakeholder  analysis,  looking  very  broadly  at  the  various 
stakeholders  and  the  desired  functions  of  EASE,  in  order  to  devise  and  prioritize  possible 
additions  or  improvements  for  the  development  team  to  include  in  future  versions. 


1.4  Study  Objectives. 

The  major  study  objectives  that  support  the  problem  statement  highlighted  above  are: 

o  Identify  M&S  community  capability  gaps 
o  Gather  hands-on  feedback  on  EASE 

o  Recommend  potential  EASE  improvements  and  enhancements 
o  Recommend  strategy  for  continued  advancement  of  EASE 
o  Develop  a  set  of  metrics  that  can  be  used  to  reflect  the  value  created  by  EASE  or 
other  M&S  initiatives 

o  Identify  potential  use  case  for  further  development 

1.5  Related  Research  and  Efforts 

MATREX.  MATREX,  developed  by  the  Army  Research  Lab  (ARL),  is  a  composable  M&S 
environment  wherein  a  collection  of  multi-fidelity  models,  simulations  and  tools  are  integrated 
into  an  established  architecture  to  conduct  analyses,  experimentation  and  technology  trade-offs 

'J 

for  the  Research,  Development,  &  Engineering  Command  (RDECOM)  and  others.  Many  of  the 
innovations  and  foundational  concepts  behind  EASE  were  first  developed  in  the  MATREX 
program. 


FACT.  The  Framework  for  Assessing  Cost  and  Technology  (FACT)  is  an  open  architecture 
web  services  based  environment,  developed  by  the  Marine  Corps  Systems  Command,  that 
enables  the  interconnecting  of  models  to  provide  a  rapid  exploration  of  the  design  tradespace  in 
support  of  systems  engineering  analysis.  FACT  is  model  agnostic  and  capable  of  linking 
disparate  models  and  simulations  of  both  government  and  commercial  origin  through  the 
application  of  community  established  data  interoperability  standards.  FACT  facilitates  rapid 


3  Hurt,  Tom,  Tim  McKelvy,  &  McDonnell,  Joe.  The  Modeling  Architecture  For  Technology,  Research,  and 
Experimentation. 
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analysis  of  alternative  technology  and  materiel  using  surrogate  models,  or  equation  regression 
representations  of  more  complex  M&S  tools.4 


C2WindTunnel.  C2  WindTunnel  is  a  software  test  bed  developed  by  George  Mason  for 
Command  and  Control  (C2)  systems.  The  software  facilitates  the  coupling  of  disparate  models 
and  simulation  engines;  enabling  the  use  of  real  world  data  across  multiple  models  expressed  in 
different  modeling  languages.  The  C2WT  framework  uses  the  discrete  event  model  of 
computation  as  the  common  semantic  framework  for  the  precise  integration  of  an  extensible 
range  of  simulation  engines.  These  simulators  are  integrated  with  the  Run-Time  Infrastructure 
(RTI)  of  the  HLA  platform.  Each  simulation  model,  when  incorporated  into  the  overall 
simulation  environment  of  C2WT,  requires  integration  on  two  levels:  the  API  level  and  the 
interaction  level. 5 


MITRE’s  Executable  Architecture.  A  MITRE  research  team  imported  key  products  of  the 
DoD  Architecture  Framework  into  an  executable  form  to  conduct  a  dynamic  analysis  of  the 
Command  and  Control  (C2)  system  or  capability  represented  by  the  architecture.  The  team 
made  a  three-way  link  between  a  business  process  model,  a  communications  network  model,  and 
a  combat  simulation  representing  the  system’s  operational  environment.  The  models  were  linked 
together  via  the  Runtime  Infrastructure  (RTI)  of  the  High  Level  Architecture  (HLA).5 6 

2.0  Methodology 

We  utilize  both  the  Systems  Decision  Process  (SDP)  and  Value-Focused  Thinking  (VFT)  to 
gather  and  analyze  stakeholder  feedback.  First,  stakeholder  perspectives,  user-types,  and 
domains  are  analyzed  to  illustrate  the  stakeholder  solution  space.  Next,  manager  interviews,  user 
surveys,  and  workshop  comments  are  gathered  to  triangulate  stakeholder  feedback  to  insure  we 
capture  the  different  perspectives,  user-types,  and  domains.  User  feedback  is  clustered  and 
organized  into  Findings,  Conclusions,  Recommendations  (FCR)  to  highlight  trends,  capability 
gaps,  and  major  issues.  The  FCR  tables  and  stakeholder  feedback  are  used  as  the  foundation  of  a 
Strengths,  Weaknesses,  Opportunities,  and  Threats  (SWOT)  analysis.  Finally,  the  SWOT 
analysis  and  stakeholder  feedback  are  translated  into  a  series  of  recommendations  regarding: 
stakeholder  solution  space  focus,  specific  M&S  organizations  with  interest,  prioritized  EASE 
improvements,  prioritized  list  of  EASE  enhancements,  and  potential  use  cases. 


4  O'Neal,  Michael.  2012.  Modeling  and  Simulation  Tool  for  Decision  Makers:  Framework  for  Assessing  Cost  and 

Technology  Project,  U.S.  Marine  Corps  Systems  Command. 

5  Roth,  Karen;  Barrett,  Shelby.  2009  (July).  Command  and  Control  Wind  Tunnel  Integration  &  Overview.  Tech. 
Report.  AFRL-RI-RS-TP-2009-14.  Air  Force  Research  Lab. 

6  Pawlowski,  Tom;  Barr,  Paul;  Ring  Steven.  2004a  (June).  Applying  Executable  Architectures  to  Support  Dynamic 

Analysis  of  C2  Systems.  Tech.  rept.The  MITRE  Corporation. 
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3.0  Stakeholder  Identification,  Categorization,  and  Gathering  Feedback. 

In  this  study,  we  conduct  a  thorough  stakeholder  analysis  to  gain  an  understanding  of  the  current 
state  of  M&S,  challenges,  capability  gaps,  and  EASE  user-level  feedback.  As  with  all  complex 
systems,  there  are  numerous  stakeholders  that  have  an  interest  in  or  are  impacted  by  M&S. 
Figure  3  highlights  the  major  stakeholders  within  the  M&S  community.  The  organizations 
highlighted  in  light  green  are  those  that  we  consulted  and  collaborated  with  throughout  the 
analysis  effort. 


tmco 


MCMSMO 


mm 


Figure  3.  Major  Modeling  and  Simulation  Stakeholders. 


3.1  Stakeholder  Classification 

Stakeholder  feedback  will  typically  vary  depending  on  the  stakeholder’s  organizational 
perspective,  specific  domain,  and  duty  position.  A  simulation  user  has  different  needs  than  a 
systems  engineer  or  manager  of  an  analytic  agency.  An  analyst  in  a  battle  lab  has  different 
combat  simulation  requirements  than  an  analyst  that  supports  collective  training  exercises. 
Specifics  about  M&S  domains,  organizational  perspectives,  and  user-types  are  discussed  in  the 
sections  that  follow. 

3.1.1  Domains. 

The  M&S  community  is  currently  organized  into  three  domains.  The  uses  of  modeling  and 
simulation  within  each  domain  vary  in  scope,  required  fidelity,  accuracy,  and  purpose.  For 
example,  a  combat  simulation  model  used  to  support  a  major  weapon  system  purchase  decision 
demands  a  higher  level  of  fidelity  and  accuracy  than  one  designed  for  individual  or  collective 
training.  Each  domain  purpose  and  associated  supporting  tasks  are  discussed  below  and 
highlighted  in  Figure  4. 
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Training  Exercises  and  Military  Operations  (TEMO).  The  primary  focus  of  the 
TEMO  domain  is  to  prepare/train  the  warfighter.  TEMO  domain  activities  include  individual 
and  collective  training,  Joint  and  combined  exercises,  mission  rehearsals,  and  operations 
planning. 

Research,  Development,  and  Acquisition  (RDA).  The  primary  focus  of  the  RDA 
domain  is  to  prepare/train  the  warfighter.  RDA  domain  activities  include  basic  applied  research, 
test  and  evaluation,  and  weapon  system  development. 

Advanced  Concepts  and  requirements  (ACR).  The  primary  focus  of  the  ACR  domain 
is  to  analyze  future  concepts  and  develop  doctrine.  ACR  domain  activities  include  force  design, 
Warfighting  experiments,  Operational  requirements,  and  analysis  of  alternatives. 


Army  M&S  Domains 


Application  Domains 


Figure  4.  M&S  Domains  and  Supporting  Tasks 


3.1.2  Stakeholder  Perspectives. 

Stakeholder  feedback  also  varies  based  upon  organizational  perspective.  The  M&S  concerns  at 
the  Enterprise  level  are  much  different  than  those  at  the  Program  level.  For  example,  Enterprise- 
level  concerns  are  focused  on  M&S  policy  and  creating  efficiencies  across  the  Enterprise.  At  the 
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program-level  concerns  are  focused  on  analytical  soundness  and  credibility.  The  three  M&S 
perspectives  are  discussed  in  more  detail  below.7 

Program.  Stakeholder  organizations  that  would  be  considered  Program- level  include: 
Maneuver  Support  Battle  Labs,  TRADOC  Analysis  Centers  (TRAC),  Army  Material  Systems 
Analysis  Agency  (AMSAA),  and  AC  ATI  programs.  They  are  primarily  concerned  with 
affordability,  credibility,  analytic  soundness,  interoperability  and  portability. 

Community.  The  planning,  testing,  training,  acquisition,  analysis,  and  experimentation 
communities  would  include  such  organizations  as  Defense  Office  of  Test&  Evaluation 
(DOT&E),  Joint  Forces  Command  (JFCOM),  Army  Modeling  and  Simulation  Working  group 
(AMSWG),  and  Program  Analysis  and  Evaluation  (PAE).  Community  stakeholders  are 
primarily  concerned  with  managing  M&S  within  their  respective  areas. 

Enterprise.  Stakeholders  that  would  be  considered  Enterprise-level  include:  Service 
Modeling  and  Simulation  Coordination  Offices  (MSCO),  OSD  Research,  Development  & 
Experimentation  (RDE),  M&S  planners  and  Integrated  Planning  Teams  (IPTs).  The  primary 
focus  at  the  Enterprise  level  includes  policy,  planning,  standards,  M&S  management,  and 
collaboration  across  communities  and  programs. 

3.1.3  Stakeholder  User-Types. 

Stakeholder  feedback  will  also  vary  depending  on  the  stakeholder’s  duty  position  or  user-level. 

A  simulation  user  has  different  needs  than  a  systems  engineer  or  manager  of  an  analytic  agency. 

User.  The  typical  M&S  user  would  utilize  combat  simulation  models  and  tools  on  a 
daily  or  weekly  basis  in  the  performance  of  their  duties.  The  user  is  primarily  concerned  with 
effectiveness,  maintainability,  applicability,  re-usability,  and  interoperability. 

Developer.  A  developer  creates,  modifies,  and  maintains  the  organizations  combat 
models  and  tools.  A  developer’s  primary  concerns  might  include  ease  of  maintenance,  updates, 
and  modifications. 

Systems  Engineer.  A  systems  engineer  develops  the  appropriate  architectures  and 
interfaces  to  facilitate  addition  and/or  federation  of  new  models  or  tools. 

The  graphic  in  Figure  5  below  highlights  the  stakeholder  solution  space  which  includes  domains, 
perspectives,  and  user-types. 


7  Aegis  Technologies,  pp  39-42. 
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Figure  5.  Stakeholder  Solution  Space 


3.2  “Triangulating”  Stakeholder  Feedback. 

We  utilized  a  combination  of  feedback  techniques  to  triangulate  stakeholder  feedback. 
Techniques  included  manager  interviews,  a  workshop,  and  a  user-level  survey  as  highlighted  in 
Figure  6  below.  The  combination  of  techniques  allowed  us  to  gather  feedback  from  the  differing 
perspectives,  user-types,  and  domains. 


Manager 

Interviews 


Hands-On 

Feedback 


Figure  6.  Methods  to  Gather  and  Triangulate  Feedback 


3.2.1  Manager  Interviews. 

A  series  of  interviews  were  conducted  with  M&S  managers  from  the  following  organizations: 
Defense  Threat  reduction  Agency  (DTRA),  Army  Material  Systems  Analysis  Agency 
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(AMSAA),  Modeling  and  Simulation  Coordination  Office  (MSCO),  Department  of  Systems 
Engineering,  and  Army  capabilities  Integration  Center  (ARCIC).  The  interviews  were 
conducted  either  by  phone  or  in  person  and  the  series  of  questions  asked  are  included  in 
Appendix  F.  Interview  responses  were  used  to  develop  the  workshop  objectives  and  refine  user 
survey  questions. 


3.2.2  EASE  Workshop. 

The  second  method  of  gathering  EASE  user-level  feedback  was  a  workshop  hosted  in  Orlando 
from  24-26  October.  The  objectives  of  the  workshop  were  to:  Expose  M&S  community 
representatives  to  the  EASE  prototype  and  the  technological  concept  behind  it  (Executable 
Architecture),  Gather  participant  feedback  on  the  EASE  prototype  and  M&S  in  general,  Identify 
organizations  with  the  most  interest  in  utilizing  EASE  and  participating  in  its  future 
development,  Prioritize  EASE  improvements  and  future  extensions,  and  Provide  a  forum  for 
M&S  user-level  professionals  to  collaborate. 

.  The  organizations  listed  below  sent  representatives  to  participate. 


FT  Benning  MTC 

FT  Stewart  MTC 

USMA 

FT  Hood  MTC 

ARCIC 

AMSAA 

FT  Campbell  MTC 

TRAC-FLVN 

TENA 

Each  participant  presented  a  mini  organization  brief  that  highlighted  their  organization’s  mission, 
M&S  challenges,  and  other  topics.  A  copy  of  the  organization  briefs  can  be  found  in  Appendix 
A.  Additionally,  participants  received  an  overview  of  EASE  and  its  capabilities,  utilized  EASE 
to  modify  and  run  a  scenario,  and  were  asked  to  provide  EASE  feedback  and  recommendations 
for  improvement/enhancement. 


3.2.3  User-Level  Survey. 

A  user  -level  survey  was  developed  and  distributed  to  the  modeling  and  simulation  community. 
Efforts  were  made  to  ensure  that  all  M&S  domains  participated  in  the  survey.  The  survey  was 
accompanied  by  a  short  demo  video  of  EASE  to  facilitate  feedback  on  the  value  of  the 
fundamental  concepts  behind  EASE.  Survey  topics  included:  frequency  and  importance  of 
M&S,  survey  of  M&S  tools  used,  duration  of  typical  modeling  and  simulation  events,  manpower 
requirements,  M&S  tool  characteristics  and  their  importance,  scenario  development  and 
modification,  and  EASE  feedback.  The  survey  instrument  can  be  found  in  Appendix  B. 


12 


4.0  Stakeholder  Feedback  Analysis. 


The  manager  interview,  workshop,  and  survey  feedback  were  analyzed  to  identify  common 
themes  and  points  of  emphasis.  The  major  takeaways  from  each  feedback  mechanism  are 
highlighted  below.  The  results  are  consolidated  and  organized  into  major  findings,  conclusions, 
and  recommendations  and  are  highlighted  in  Section  4.4  and  included  in  Appendix  G. 

4.1  Manager  Interview  Feedback 

Not  surprisingly,  managers  were  concerned  about  slightly  different  issues  than  analysts  and 
simulation  users.  The  key  points  from  the  manager  interviews  are  discussed  below. 

Inefficiencies  and  Duplication.  Managers  acknowledged  that  there  are  numerous 
redundancies  in  the  models  and  tools  used  both  within  and  across  M&S  domains.  Additionally, 
there  is  no  capability  to  quickly  access  and  leverage  the  myriad  of  models  used  throughout  the 
community.  They  agree  that  some  well  managed  central  repository  could  provide  value  to  the 
M&S  community. 

No  real  centralized  management.  Managers  voiced  significant  concern  with  the  lack  of 
centralized  planning,  strategy,  guidance,  and  synchronization  of  major  efforts  that  impact  the 
M&S  domains.  They  note  that  MSCO  and  AMSO  have  made  recent  attempts  to  improve 
planning  and  synchronization  but  highlight  that  most  efforts  have  fallen  short. 

VV&A.  Managers  expressed  hesitation  in  adopting  EASE  or  other  new  tools  due  to 
restrictions  requiring  use  of  only  “W&A”  models  although  they  acknowledge  that  W&A  has  a 
very  loose  definition  and  vague  standards. 

Budget.  Given  the  current  economic  environment,  managers  welcome  any  effort  that 
could  reduce  their  hardware  and  software  footprint  and  decrease  manpower  dedicated  to 
managing  them.  They  highlighted  the  significant  resources  devoted  to  routing  maintenance  and 
updates. 


4.2  Workshop  Feedback 

After  familiarizing  with  the  concepts  behind  EASE  and  a  day  of  hands-on  EASE  application, 
workshop  participants  were  asked  about  general  M&S  limitations,  strengths  of  EASE, 
weaknesses  of  EASE,  recommended  EASE  enhancements,  and  level  of  interest  in  EASE. 

M&S  Limitations.  The  M&S  limitations  highlighted  by  workshop  participants  are  very  similar 
to  those  provided  by  survey  respondents  (discussed  in  Section  4.3  below).  The  most  common 
limitations  mentioned  by  workshop  participants  were: 
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Terrain.  There  is  a  lack  of  common,  standardized,  correlated  terrain.  Additionally, 
terrain  formats  and  accuracy  vary  from  model  to  model.  Lastly,  there  are  several  important 
geographic  regions  that  lack  sufficient  terrain  data  suitable  for  M&S  use. 

Interoperability.  Because  there  is  not  a  combat  simulation  model  that  meets  everyone’s 
needs  yet,  there  is  functionality  in  many  models  that,  if  federated  with  other  models  could  greatly 
enhance  research  and  analysis  efforts.  However,  integrating  disparate  models  takes  tremendous 
time,  expertise,  and  manpower. 

Training  Requirements.  Most  combat  simulation  models  and  tools  require  a  significant 
amount  of  training  to  just  become  proficient  at  a  basic  level.  Most  day-to-day  research  and 
analysis  work  requires  a  much  higher  level  of  proficiency  in  manipulating  individual  models  and 
integrating  them  with  other  models,  tools,  or  scripts. 

Execute  Rapid  Changes.  Overly  complex  combat  simulation  software  and  intricate 
interfaces  make  minor  model  or  scenario  changes  non-trivial  and  time  consuming.  Even  re¬ 
running  previous  experiments  is  problematic  because  of  ever  changing  software  and  hardware 
configurations  and  profiles. 


Figure  7  below  highlights  the  workshop  participant  organization  mission  and  their  biggest  M&S 
limitations. 
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Organization 

Mission 

Biggest  M&S  Shortfalls 

Ft,  Hoed  Mission  Command 

Training  Center 

Support  unit  training  objectives 
using  the  Army's  Mission 

Command  Systems 

Maintaining  relevancy  with  the  training  unit  requirements.  Changes  in 
MC5.  Scenarios  (terrain,  equipment). 

Ft.  Banning  Maneuver 

Battle  Lab 

Recommend  DOTMLPF  solutions 

based  on  LVCAR&G  experiments 

inability  to  provide  an  integrated,  comprehensive  Common  Operating 
Picture  (COP)  to  all  Joint  Mission  Command  Systems  in  the  current 
Simulation  Architecture.  Applies  to  Air  and  Missile  Defense 

Workstation  (AMDWS),  Forward  Area  Air  Defense  -  Engagement 

USMA  Department  of 
Systems  Engineering 

Educate  West  Point  cadets 

Technical  knowledge  and  talent  availability  /  reliance  on  small  staff. 

Ft,  Stewart  Mission  Training 
Complex 

Training  and  facility  support  to 
units  in  LVC  simulation  exercises 

at  all  levels 

M&S  Terrain  availability  to  match  terrain  used  in  Mission  Command 
systems.  Interoperability  across  all  M&S  systems. 

Ft*  Leavenworth  TRADOC 

Analysis  Center  [TRAC) 

Operations  analysis  to  inform 
decisions  across  the  spectral 
(conepts  to  operations) 

Mission  Command  and  Cultural  Effects 

Army  Capabilities 

Integrate  on  Center  f  ARCtC} 

Development  and  integration  of 
force  capabilities  across  the 
DOTMLPF 

Representing  all  current  and  future  Warfighting  Functions  accurately 
enough  (entity  level)  while  addressing  higher  echelon  concepts  (Div 
&JTF) 

Test  and  Training  Enabling 
Architecture 

Architecture  and  software 
necessary  to  enable  testing  and 
training  on  ranges 

Lack  of  full  Suite  of  object  models.  DDM  support. 

Ft*  Campbell  Mission 
Training  Complex 

Mission  Command  digital  training 
for  mylti- echelon  combined  arms 
operations 

Scenario  generation  timelines.  Formal  system  training.  Funding 
challenges  With  respect  to  class  availability  Such  as  terrain  building 
and  scripting,  integration  between  current  simulations 

Army  Materiel  Systems 
Analysis  Activity  (AMSAA) 

Conduct  analyses  across  Materiel 
Life  Cycle  informing  Army 
decisions 

Maintaining  supporting  infrastructure  fora  broad  array  of  specific 
systems'  operating  requirements,  data  storage  and  retrieval,  and 
search  engine  to  enable  study  development  and  M&S  tool  upgrades 

USAOTC  Test  Technology 
Directorate  (TTD) 

Support  for  test  directorates  in 
execution  of  operational  test  in 
joint  environments 

Monfunctional  analysis  (e,g.,  performance,  scalability)  of  M&S  system 
of  systems  in  relation  to  system  under  test* 

Figure  7.  Workshop  Participant  Organizational  M&S  Shortfalls 


EASE  Strengths.  Figure  8  below  highlights  EASE  strengths  identified  by  workshop 
participants.  Most  notably,  participants  were  not  aware  of  another  “EASE-like”  product  in  use 
or  under  development.  They  value  EASE  capability  to  document  and  archive  model  architecture 
and  interoperability  requirements.  Workshop  participants  see  the  ability  to  maintain  and  reuse 
previous  combat  simulation  scenarios  and  runs  as  a  clear  strength.  The  surrogate  capability 
provided  in  EASE  was  highlighted  as  unique  and  positive.  Lastly,  workshop  participants  note 
that  EASE  has  potential  to  both  reduce  their  hardware  and  software  footprint  as  well  as  provide  a 
back-up  capability. 


15 


EASE  Positives  orqetu 

Isjn  retail 

+  Architecture  Capability/Capture 
+  Capture  Interoperability  Requirements 
+  Capability  Focused  versus  Application  Focused 
+  Captures  Knowledge 
+  M  a  inta  ins  Repository  of  M &S 

+  Could  Simplify  Configuration  and  Execution  for  User 
+  May  Help  Administrator  Role 
+  Test  Capability  and  Repeatability 
+  Ability  to  Federate  to  Fill  Model  Gaps 
+  Can  Assist  with  Configuration  Management 
+  Redundant  Capability  to  Current  Hardware  Footprints 

2H2  LEBT  WcrfahcFi  2-3 -JK  Ot'  !_Z 

Figure  8.  EASE  Strengths  Identified  by  Workshop  Participants 

EASE  Weaknesses.  Figure  9  below  highlights  EASE  weaknesses  identified  by  workshop 
participants.  Workshop  participants  were  concerned  about  having  a  lack  of  in-house,  EASE 
expertise;  potentially  creating  a  single  point  of  failure.  They  note  that  they  lack  the  manpower, 
expertise,  and  experience  required  to  build  SDDs  that  would  properly  function  within  EASE. 
Because  only  a  few  models,  scenarios,  and  supporting  SDDs  are  currently  represented  in  EASE, 
workshop  participants  feel  that  EASE  would  not  provide  any  additional  advantage  over  the 
current  way  of  doing  M&S  business  and  would  make  it  difficult  to”  sell”  to  their  managers  and 
fellow  M&S  users.  Lastly,  the  perceived  risk  associated  with  not  being  a  Program  of  Record 
(POR)  was  highlighted  as  a  major  weakness. 
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EASE  Weaknesses  one 

O  5 

-  Potential  Single  Point  of  Failure 

-  Entity  Mapping  and  Translation  (WA) 

-  Building  SDDs... Capacity  and  QC 

-  Initial  Feed  the  Dragon 

-  "Not  yet  Customized  to  Us" 

-  Limited  Current  Application-No  Additional  Advantage 
•Could  Not  Replicate  Current  Hardware  (600  machines) 
-Questionable  Whether  Long  Term  Solution  ...Not  POR..Risk 


ERSE") 


SUE  LB-'  WiHkJlofK  M-Ji  QcT  13 


Figure  9.  EASE  Weaknesses  Identified  by  Workshop  Participants 


Recommended  EASE  Enhancements.  Figure  10  below  highlights  recommended  EASE 
enhancements  identified  by  workshop  participants.  EASE  enhancements  were  binned  by  priority 
as  recommended  by  workshop  participants.  A  common  topic  throughout  the  workshop  and  a 
recommended  high  priority  enhancement  is  integration  of  EASE  with  current  Mission  Command 
and  C2  Systems.  Workshop  participants  also  recommended  adding  many  more  scenarios, 
models,  and  supporting  SDDs  for  the  most  commonly  used  combat  simulation  models  and  tools. 
An  EASE  linkage  to  Force  Builder  was  mentioned  as  a  medium  priority  recommended 
enhancement.  Low  priority  recommended  enhancements  include  a  robust  report  and  analysis 
capability  and  linkage  of  terrain  to  the  application  line-up. 
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Recommended  EASE  Modifications 

R  5  and  Extensions 

Modifications  and  Extensions 

Priority 

Integrate  w/  Mission  Command  Systems 

Enumeration  and  Comparison  Capability 

Configure  and  Run  Big  Model— Small  Model 

SDDs  for  Current  Common  Tools 

HIGH 

Link  to  Web-MSDE 

Link  to  Force  Builder->  MSDL 

Parametric  Data  Linkage 

Database  Builder 

MEDIUM 

Tie  Terrain  to  Line-Up 

Measurement  Space  GUI 

Leverage  Existing  Network  Resources 

Back  End  Reports  and  Analysis 

LOW 

EASE  USER  Workshop:  24-26  Oct  12 

Figure  10.  Recommended  EASE  Enhancements  from  Workshop  Participants 


Level  of  Interest  in  EASE.  Workshop  participants  were  asked  to  quantify  their  organization’s 
level  of  interest  in  using  EASE.  Figure  1 1  below  highlights  the  continuum  of  workshop 
participant  interest  in  EASE.  Those  highlighted  with  an  asterisk  had  interest  in  only  portions  of 
EASE.  FT  Benning  MTC  has  the  highest  level  of  interest  in  EASE  and  its  unique  capabilities. 
FT  Hood  MTC,  USMA,  and  TENA  have  mild  to  strong  interest.  TRAC  and  ARCIC  have  the 
least  amount  of  interest  in  EASE.  Their  hesitancy  is  primarily  attributed  to  a  lack  of  EASE 
W&A  credentials  and  applicability  to  their  day-to-day  work. 


No  Passive  Strong  When  can 

Interest  Interest  Interest  we  start? 


TRAC 


♦TRAC-FLVN 

*  Stewart  MTC  *Benning  MTC 
AMSAA 


♦Campbell  MTC 

Hood  MTC 

ARCIC  TENA 

USMA 


Figure  11.  Workshop  Participant  Interest  in  EASE 


18 


4.3  User  Survey  Feedback. 

As  highlighted  earlier,  a  survey  was  developed  and  distributed  to  the  TEMO,  ACR,  and  RDA 
M&S  communities.  Approximately  100  M&S  users  from  across  all  domain  responded  to  the 
survey.  The  survey  instrument  is  included  at  Appendix  B.  A  complete  description  of  all  survey 
results  is  included  in  Appendix  C,  however  the  key  points  from  the  survey  are  discussed  below. 

Frequency  and  importance  of  M&S.  Survey  respondents  were  asked  to  quantify  how 
frequently  the  utilized  M&S  in  the  course  of  their  work  and  the  importance  of  M&S  relative  to 
other  tolls  or  techniques.  71%  of  respondents  stated  that  they  use  M&S  on  a  daily  basis  and  over 
90%  classify  M&S  as  important  or  very  important  to  the  work  of  their  organization. 


Frequency  of  Simulation  Use 


Daily  Weekly  Monthly  Few  Once  or 
Times  Twice  a 
Per  year  Year 


Frequency 


Importance  of  Simulation 


70  - 6S3& 


Not  Somewhat  Important  Very 
Important  Important  important 

Importance 


Figure  12.  Frequency  and  Importance  of  Simulation  Use 


Most  common  combat  simulation  models  and  tools.  Survey  respondents  were  asked  to 
list  combat  simulation  models  and  tools  they  use.  As  expected,  there  are  a  wide  variety  of  tools 
used  across  and  within  M&S  domains.  The  top  four  combat  simulation  software  packages  used 
by  survey  respondents  were  JCATS,  VBS2,  OneSAF,  and  Combat  XXI. 
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Top  4  Combat  Simulation 
Software 


JCATS  VBS2  OneSAF  Combat  XXI 


Commbat  Simulation  Software 


Figure  13.  Top  Combat  Simulation  Software  Used 


Current  M&S  limitations.  Respondents  were  asked  to  list  the  limitations  of  current  combat 
simulation  models  and  tools.  The  most  common  limitations  listed  in  priority  order  include: 

•  Validity  and  available  of  data 

•  Interoperability  with  other  models  and  tools 

•  Trained  users 

•  Lack  of  tools  to  quickly  create  or  modify  a  scenario 

•  Keeping  hardware  and  software  current 

•  Common,  correlated  terrain 

•  Cumbersome  user  interfaces 


Typical  duration  of  M&S  event.  Survey  respondents  were  asked  to  quantify  the  time 
typically  spent  during  each  phase  of  a  simulation  event;  Systems  Engineering,  Development, 

Data  Engineering,  Testing,  Execution,  and  Analysis.  Survey  respondents  reported  that  Testing, 
Execution,  and  Analysis  phases  take  the  least  amount  of  time  lasting  between  2  days  and  2 
weeks.  The  Development  and  Systems  Engineering  phases  are  typically  the  longest.  The 
Development  phase  is  generally  greater  than  2  weeks  in  duration  and  has  the  greatest  variation  of 
the  M&S  event  phases. 
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Duration  of  Testing  Phase 


Duration  of  Analysis  Phase 


Duration  of  Execution  Phase 


<2  days  2day$-2  2-6  Weeks  >6  Weeks 
Weeks 


Duration 


<2  days 


2  days-2  2-6  Weeks  >6  Weeks 
Weeks 


Duration  of  Systems  Engineering 
Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 


Duration  of  Development  Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 

Duration 


Duration  of  Data  Engineering 
Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 


Duration 


Figure  14.  Duration  of  Simulation  Event  Phases 


Verification,  Validation,  and  Accreditation  (VV&A).  Respondents  were  asked  about 
the  importance  of  utilizing  only  models  and  tools  that  were  VV&A  as  well  as  the  proportion  of 
their  organization’s  models  and  tools  that  are  actually  VV&A. 


VV&A  Importance 


Not  Somewhat  Important  Show 
Important  Important  Stopper 

Importance 


Models  and  Tools  That  are 
VV&Ad 


0-25  25-50  50-75  75-100 


Percentage  of  Tools 


Figure  15.  VV&A  Importance 
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M&S  important  characteristics.  Survey  respondents  were  asked  to  rate  the  importance 
as  well  as  the  relative  comparison  of  the  characteristics  listed  below. 

•  Low  barrier  to  use  for  varying  M&S  skill  levels 

•  Ability  to  access  and  run  from  anywhere 

•  Integrates  with  other  commonly  used  M&S  packages  and  scripts 

•  Customizable  output  graphics  and  statistics 

•  Front  end  DOE  capability  to  plan  and  customize  experiments 

•  Ability  to  add  and  modify  scenarios 

•  Ability  to  interface  and  draw  from  authoritative  data  sources 


Importance  of  Integrating  and  Drawing  from 
Authoritative  Data  Sources 


Add  and  Modify  Scenario 
Importance 


Level  of  Importance  (l=Low,  7=High) 


Level  of  Importance  (l=Low,  7=High) 


Importance  of  Integrating  with 
other  M&S  Models  or  Tools 


1  2  3  4  5  6  7 

Level  of  Importance  (1-Low,  7=High) 


Figure  16.  Comparison  of  Important  M&S  Criteria 


The  ability  to  add  and  modify  scenarios  and  integrate  with  other  M&S  tools  were  highlighted  as 
the  most  important  characteristics.  The  ability  to  access  and  run  from  anywhere  and  a  front  end 
DOE  capability  to  plan  and  customize  experiments  were  rated  as  the  least  important  of  the 
criteria. 
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Pre  and  Post-processing  tools.  Survey  respondents  were  asked  to  list  pre  and  post¬ 
processing  tools  they  use.  Again,  there  is  a  wide  variety  of  tools  used  across  and  within  M&S 
domains.  The  tables  below  highlight  the  various  pre  and  post  processing  tools  used  by  the 
survey  respondents. 


Pre-processing 


Order  Of  Battle  Serves  (QBS)JLCCTC  Validational  Tools 

Joint  Training  Data  Services 

JTDS,  Joint  Remote  Client 

APE:  AWARS  Pre-processing  Environment 

IPR,  MSELSync  meeting,  datbase  creation,  RTOC  setup 

LOD 

Polaris 

A  MS  A  A  Joint  Data  Center  (JDC) 

SAF 

SIMPLE 

COMBATXXI  Preprocessor  (AKA  Scenario  Integration  Tool  Suite} 

AWARS  uses  two  preprocessors:  Ape  and  FSMP. 

SQL,  Excel 

Electronic  Data  Request  System  (eDRS) 

ArcGIS,  TerraSim  TerraTools 

Open  Office,  data  tools  include  MS  Access  and  MS  Excel 

Post- processing 


After  Action  Review  System  (AARSJ  that  is  part  of JLCCTC  ERF 

AWARS  Post-processor,  SAS 

AAR'S,  ICE  FORMS 

LOD 

GUI  standard  output  packages 

IMSITE 

In-house-developed  data  reduction  software 

Microsoft  Excel,  Microsoft  Word,  Microsoft  SQL  Server 

Custom  SQL  Server-based  post  processor.  Excel. 

SAF/AAR 

PASW  (AKA  Clementine),  Excel,  and  SQL. 

SQL  Data  base  and  Excel. 

Figure  17.  Pre  and  Post  Processing  Tools  Commonly  Used 


Scenarios.  Survey  respondents  were  asked  how  frequently  scenarios  are 
adjusted/updated,  how  often  new  scenarios  are  developed,  and  the  time/manpower  requirements 
typically  required  to  develop  and  modify  scenarios.  The  majority  of  survey  respondents  stated 
that  new  scenarios  are  typically  developed  monthly  or  a  few  times  per  year-  most  often  requiring 
a  minimum  of  2  to  3  scenario  developers.  25%  of  organizations  noted  that  they  require  more 
than  5  scenario  developers  to  generate  a  new  scenario.  Recall  from  the  earlier  survey  results 
discussion,  the  ability  to  modify  and  develop  scenarios  was  highlighted  as  the  most  desirable 
characteristic.  Finally,  feedback  highlighted  in  Figure  19  illustrates  that  there  is  a  wide  variety 
of  scenario  formats  used  throughout  the  M&S  community. 


23 


Frequency  of  New  Scenario 
Development 


Times  a  Twice 
year 


Frequency 


Criteria  Comparison 

70 


Cate  gory/C  rite  ri  an 


Number  of  Scenario  Developers 


>2.08 

5 

16.67 

ft 

■ 

1  2-3  4-5  >5 

Number  of  Developers 


Figure  18.  Add  and  Modify  Scenario  Importance 


Scenario  Formats 

Save  scenarios  in  each  federate  [VB32,  QneSAF,  etc...) 

QBS  V2.0 _ 

MS  Powerpoint/Word,  Adobe  Reader,  C2PC  (Command  and  Control  Personal  Computer) 
CD _ 

Disks,  Sharepoint:, 

VB52  Format _ 

QneSAF  -  XML  (BZIP'dj _ 

SAF _ 

JCATSVBS2 _ 

RTF  files  or  xml _ 

XML _ 

XML  and  odb  (open  office  database). 

Word  and  PowerPoint. 


Figure  19.  Scenario  Formats  Commonly  Used 


EASE  feedback.  Based  upon  a  brief  video  demo  of  EASE,  survey  respondents  were 
asked  to  comment  on  the  usefulness  of  EASE  and  highlight  features/functionality  that  would 
increase  the  likelihood  that  they  would  use  EASE.  Results  are  highlighted  in  Figure  20  below. 
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Functionality  that  would  increase  liklihood  of  EASE  use 

Easy  to  use  information  exchange  and  data  mapping. 
Interoperability  with  Army  Mission  Command  Systems _ 

Simplicity  and  the  ability  to  customize  scenarios  to  meet  unit 
training  objectives 

Being  User  Friendly _ 

Ability  to  incorporate  standard  data  products.  Ability  to 
integrate  with  other  sim  tools. _ 

Confidence  that  the  data  provided  is  accurate  and  can  be 
understood  by  an  outside  agency. 

AAR  info _ 

Providing  interaction  forSoldiers  that  reduce  the  resource 

requirement  but  allows  a  full  range  of  usage 

Must  be  user-friendly,  handle  classified,  be  V&V'd _ 

Automatic  configuration  and  Launching. 

Native  support  forTENA 

Easy  to  use  execution  and  analytical  tools  for^IOO  replications 
for  each  alternative  within  DoD  compliant  and  approved 


Figure  20.  Desirable  EASE  Features 


4.4  Findings,  Conclusions,  Recommendations  (FCR). 

Complete  FCR  tables  are  included  in  Appendix  G.  Key  FCR  threads  are  discussed  briefly 
below. 


Scenarios.  The  ability  to  modify  and  develop  scenarios  was  highlighted  as  the  most 
desirable  characteristic  by  survey  respondents.  The  majority  of  survey  respondents  stated  that 
new  scenarios  are  typically  developed  monthly  or  a  few  times  per  year-  most  often  requiring  a 
minimum  of  2  to  3  scenario  developers.  25%  of  organizations  noted  that  they  require  more  than 
5  scenario  developers  to  generate  a  new  scenario.  User  feedback  also  indicated  that  there  is  a 
wide  variety  of  scenario  formats  used  throughout  the  M&S  community.  Workshop  participants 
expressed  that  rapid  scenario  modification  and  development  were  very  desirable  features  of  any 
combat  simulation  model  or  tool. 

Data.  Survey  respondents  identified  the  ability  to  integrate  with  and  draw  from 
authoritative  data  sources  as  a  top  three  desirable  characteristic.  Workshop  participants  note  that 
requested  data  often  takes  months  to  arrive.  Additionally,  significant  amounts  of  time  are 
devoted  to  validating  this  data  prior  to  use.  Often  times  there  are  compatibility  issues  with  data 
used  with  multiple  models.  Terrain  data  is  often  lacking  for  particular  geographic  regions  and  it 
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is  also  not  always  compatible  across  models.  We  recommend  adding  terrain  data  sets  to  EASE 
within  the  line-up  and  providing  the  capability  to  integrate  with  common  data  sources. 

Hardware  and  Software  Footprint.  Each  combat  simulation  lab  or  analysis  center 
maintains  computers,  servers,  and  specialized  equipment  to  support  M&S  for  experimentation, 
training,  and  analysis.  FT  Stewart  MTC  maintains  a  53K  sq  ft  facility  and  35  servers.  The 
maneuver  battle  lab  at  FT  Benning  maintains  a  120,000  Sq.  Ft.  Constructive/ Virtual/ Gaming 
Simulation  Facility  with  450  desktop  computers  and  50  servers.  These  resources  are  replicated 
at  each  battle  lab,  MTC,  and  analysis  center  at  great  expense.  Additionally,  each  battle  lab, 
MTC,  and  analysis  center  maintains  a  massive  suite  of  combat  simulation  software.  A  cloud  - 
based  solution  has  the  potential  to  provide  substantial  cost  savings  across  the  M&S  Enterprise. 

Interoperability/Integration.  With  the  fielding  of  new  equipment  and  new  threats, 
additional  combat  simulation  models  and  tools  are  required.  These  new  models  and  tools  are 
not  necessarily  intended  to  work  together  however  it  is  highly  desirable  to  integrate  multiple 
models.  Stakeholders  identified  integration  of  multiple  models  or  tools  as  very  important  and  a 
top  three  characteristic.  Consequently,  EASE  extensions  should  focus  an  increasing  the 
timeliness  and  reducing  manpower  required  to  integrate  multiple  models.  Additionally,  the 
application  line-up  database  and  function  vs.  application  focus  are  unique  and  positive  aspects  of 
EASE  and  should  be  leveraged. 

Reuse.  Stakeholders  noted  great  value  in  the  ability  to  access  previously  run  simulations 
without  the  burden  of  new  configuration  work  or  software  updates.  Being  able  to  access  and 
rerun  any  simulation  archived  in  EASE  was  seen  as  positive.  Additionally,  strategic  M&S 
guidance  lists  reuse  as  an  important  Enterprise  M&S  characteristic. 

VV&A.  Managers  expressed  hesitancy  in  adopting  EASE  due  to  its  lack  of  VV&A 
certification.  87%  of  survey  respondents  stated  that  VV&A  of  a  combat  simulation  model  or 
tool  was  very  important  within  their  organization.  Most  report  that  over  75%  of  the  models  and 
tools  they  use  on  a  day-to  day  basis  are  VV&Ad.  Stakeholders  in  the  TEMO  domain  expressed 
less  concern  in  VV&A  than  ACR  and  RDA  domains.  We  recommend  identifying  an  appropriate 
W&A  authority,  discuss  the  specific  VV&A  requirements  as  they  pertain  to  EASE,  and  begin 
action  on  those  W&A  related  tasks  that  can  be  completed  now. 


4.5  EASE  SWOT  Analysis 

The  user  survey,  workshop  feedback  ,  and  the  FCR  are  translated  into  a  SWOT  analysis  which  is 
highlighted  in  Figure  21  below. 
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EASE  SWOT  ANALYSIS  CiffctiM 

STRENGTHS 

WEAKNESSES 

MU  Cc  mmu  nity  not  .aware  of  a  .similar  product 

EASE  provides  value  to  several  user-typas  and  domains 

EASE  provid  es  he  listtc  so  lut'io  n  tc  M  LS  cc  rr:mc  n 
challenge  .5 

Maintains  repository  of  MLS  and  facilitate  s  reu.se 

Capability  fccu.se d  versus  application  focused 

Architecture  and  documentation  capabilities 

Provides  documentation  and  ability  tc  leve ra." 
nu  me  re  us  to  dels 

In  current  state..  EASE  has  limited  application 

Limited  ’underthe  hoed'’  understanding  of  EASE 

EASE  seen  as  s  he  rt  te  rm  sc  lutio  n  b  ecause  not  a  P  CP 

Time.. expertise.,  and  resources  required  tc  developSDDs 

OPPORTUNITIES 

THREATS 

D  e  ve  1=  p  meut  cf  SD  Ds  for  Test  cc  mma  n  ap  p licatic  ns 

Ve  rificat  to n ..  Va  1  id atio n ..  S.  Accreditation  ^WA| 

Reduce  hardware  footprint  and  manpower  through  ccud 

Hot  customized  to  us'’ 

and  virtual  rnachines 

Atte mpt to soKe  eve rything. ...therefore  sc Ive  ncthing 

Integrate  with  Mode :  i  as  e  d  Syste  n\s  Engin  e  e  ring  effc  rts 

Organizations  will  net  touch  until  program  of  record 

Seen  arse  modification  and  development  capability  Is 

Partial  transition  p  Ian  ...transition  b  ut  still  provide 

h'^gh  ly  desired 

Set  EASE  into  a  selected  !ab  and  demonstrate  benefits 

extensive  support 

Figure  21.  EASE  SWOT  Analysis 


5.0  Recommendations  and  Conclusions 

The  objectives  of  this  study  were  to:  identify  M&S  community  capability  gaps,  gather  hands-on 
feedback  on  EASE,  recommend  potential  EASE  improvements  and  enhancements,  recommend  a 
strategy  for  continued  advancement  of  EASE,  develop  a  set  of  metrics  that  can  be  used  to  reflect 
the  value  created  by  EASE  or  other  M&S  initiatives,  and  identify  a  potential  use  case  for  further 
development.  The  conclusions  related  to  each  are  discussed  below. 

Current  M&S  limitations.  The  most  common  limitations  identified  by  M&S  stakeholders, 
listed  in  priority  order  include: 

•  Validity  and  available  of  data 

•  Interoperability  with  other  models  and  tools 

•  Trained  users 

•  Lack  of  tools  to  quickly  create  or  modify  a  scenario 

•  Keeping  hardware  and  software  current 

•  Common,  correlated  terrain 

•  Cumbersome  user  interfaces 
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EASE  Hands-On  Feedback.  Workshop  participants  were  not  aware  of  another  “EASE-like” 
product  in  use  or  under  development.  They  value  EASE  capability  to  document  and  archive 
model  architecture  and  interoperability  requirements.  They  saw  the  ability  to  maintain  and  reuse 
previous  combat  simulation  scenarios  and  runs  as  a  clear  strength.  The  surrogate  capability 
provided  in  EASE  was  highlighted  as  unique  and  positive.  Lastly,  workshop  participants  note 
that  EASE  has  potential  to  both  reduce  their  hardware  and  software  footprint  as  well  as  provide  a 
back-up  capability. 

Workshop  participants  were  concerned  about  having  a  lack  of  in-house,  EASE  expertise; 
potentially  creating  a  single  point  of  failure.  They  note  that  they  lack  the  manpower,  expertise, 
and  experience  required  to  build  SDDs  that  would  properly  function  within  EASE.  Because  only 
a  few  models,  scenarios,  and  supporting  SDDs  are  currently  represented  in  EASE,  workshop 
participants  feel  that  EASE  would  not  provide  any  additional  advantage  over  the  current  way  of 
doing  M&S  business  and  would  make  it  difficult  to”  sell”  to  their  managers  and  fellow  M&S 
users.  Lastly,  the  perceived  risk  associated  with  not  being  a  Program  of  Record  (POR)  was 
highlighted  as  a  major  weakness. 


Recommended  EASE  Enhancements.  Stakeholders  recommended  integration  of  EASE  with 
current  Mission  Command  and  C2  Systems  as  a  high  priority  enhancement.  They  also 
recommended  adding  many  more  scenarios,  models,  and  supporting  SDDs  for  the  most 
commonly  used  combat  simulation  models  and  tools.  An  EASE  linkage  to  Force  Builder  was 
mentioned  as  a  medium  priority  enhancement.  Low  priority  recommended  enhancements 
include  a  robust  report  and  analysis  capability  and  linkage  of  terrain  to  the  application  line-up. 


EASE  Strategy.  Our  recommended  EASE  strategy  includes  Who  and  Where  to  focus  future 
EASE  efforts.  Because  the  stakeholder  solution  space  is  so  diverse,  a  tool  that  attempts  to  solve 
every  problem  will  collapse  under  its  own  weight.  Based  upon  workshop  feedback  and 
stakeholder  solution  space  analysis  we  recommend  the  EASE  development  team  focus  its  efforts 
on  the  Program-RDA-User-SE-Developer  portion  of  the  stakeholder  solution  space  as 
highlighted  in  Figure  22  below. 
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Figure  22.  Recommended  Stakeholder  Space  Focus 


The  high  priority  collaborative  EASE  partners  include  MSCoE,  USMA,  and  FT  Benning  MTC 
since  they  have  the  highest  Interest/Potential  Applicability  as  highlighted  in  Figure23  below. 


Figure  23.  High  Priority  EASE  Collaborators 


In  general,  prioritized  EASE  improvements  should  focus  on  scenario  development/modification 
capability,  increasing  the  ease  of  integrating  disparate  models,  establishing  linkages  to 
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authoritative  data  sources,  and  continuing  to  populate  the  application  database  with 
accompanying  SDDs.  The  EASE  development  team  can  assist  the  M&S  community  the  most  by 
focusing  improvements  and  enhancements  on  efforts  that  provide  value  to  the  Systems 
Engineering  and  Development  phases  of  the  M&S  lifecycle.  We  recommend  adding  VBS2, 
Night  Vision  Tool  Kit,  and  JCATS  to  the  application  database  and  line-up  with  appropriate 
supporting  SDDs. 

A  final  general  recommendation  is  to  identify  an  appropriate  VV&A  authority,  discuss  the 
specific  VV&A  requirements  as  they  pertain  to  EASE,  and  begin  action  on  those  VV&A  related 
tasks  that  can  be  completed  now.  Figure  24  below  highlights  the  recommended  areas  of  focus. 
Highest  priority  should  be  given  to  EASE  enhancements  that  are  highlighted  in  green. 


Stakeholder 


Applications 

VBS2 
NVTK 
JCATS 
Combat  XXI 
FIRESIM 


Figure  24.  Recommended  EASE  Enhancement  Focus 


We  recommend  two  use  cases  to  demonstrate  the  valuable  and  innovative  capabilities  of  EASE: 

MSCoE.  Utilize  EASE  to  support  their  upcoming  SIMEX.  EASE  can  improve  MSBL 
execution  of  simulation  both  in  the  short  term  as  well  as  the  long  term.  In  the  short  term,  EASE 
could  facilitate  the  automation  of  execution  of  M&S  across  their  lab  assets.  EASE  would  capture 
the  technical  complexity  of  their  simulation  environment  and  provide  a  simple  interface  to 
execute  M&S  as  well  gather  AAR  products  through  a  single  web  interface.  In  the  long  term, 
EASE  could  be  used  to  link  simulation  capabilities  with  low  level  technical  design  details.  This 
will  ultimately  lead  to  better  reuse  and  interoperability  providing  cheaper  and  more  accurate 
MSBL  M&S  usage. 
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USMA.  Utilize  EASE  to  facilitate  DSE  work  in  support  of  their  Squad  X  and 
Deployable  Force  Protection  (DFP)  projects.  Specifically,  use  EASE  to  help  develop  system  of 
system  federations  that  support  each  program.  Key  capabilities  required  will  be  systems 
engineering  analysis,  federation  management  and  start/stop,  and  data  collection.  DSE  would  like 
to  assess  EASE  ability  to  build  command  and  control  data  models  and  simulation  federates  that 
pass  federation  data  to  command  and  control  systems  used  for  both  DFP  and  Squad  X. 
Additionally,  once  loaded  in  DSE  labs,  EASE  could  be  used  to  support  the  combat  simulation 
and  architecture  courses. 


5.1  Metrics 

Aegis,  in  their  work  entitled  “Metrics  for  Modeling  and  Simulation  (M&S)  Investments”  conduct 
an  extensive  analysis  of  potential  metrics  for  M&S  investments  and  return  on  Investment  (ROI). 
Metrics  were  developed  for  each  of  the  multiple  user  perspectives  (i.e.  Enterprise,  Community, 
Program,  etc.),  and  address  both  quality  and  monetary  aspects.  Below  is  a  value  hierarchy  that 
synthesizes  stakeholder  feedback  and  selected  AEgis  metrics8.  Metrics  are  defined  in  Appendix 
H. 


Figure  25.  EASE  Value  Hierarchy  and  Metrics 


8  AEgis  Technologies  Group,  Inc.  2008  (November2008).  Metrics  for  Modeling  and  Simulation(M&S)  Investments. 
Tech.  rept.  Report  NumberTJ-042608-RRP013.  AEgis. 
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Appendix  B.  EASE  Survey 


EASE 

Page  3  of  7 


Adminstrative  Information 

1.  How  many  years  of  experience  do  you  have  using  combat  simulations?* 

1-3  Years 

2.  Which  best  describe  your  current  role  as  it  relates  to  M&S.* 

Select  up  to  tw  o  choices 

□ Simulation  User 

□  Manager  of  Simulation  Users 

□  Simulation  Developer 
Simulation  Data  Provider 

□  Systems  Engineer 

□  Scenario  Developer 

□  Federation  Integrator 

3.  Please  select  your  organization  from  the  list  below:* 

Other 

4.  If  your  organization  was  not  listed  in  item  3  above  and  you  selected  "other"J  please  enter  it  below, 


Back 


Next 


Cancel 
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EASE 


Page  4  of  7 


Organization  Simulation  Use 


Based  on  your  Simulation  experience  in  your  current  organization.... 


5.  How  frequently  does  your  organization  use  simulation? 


Frequency  of 
Simulation  Use 


A  Few 

Once  or 

Week  ly 

Monthly 

times  per 

Twice  a 

Year 

Year 

© 

© 

© 

© 

6.  How  would  you  classify  the  importance  of  combat  simulation  to  the  accomplishment  of  your  organization's  day  to 
day  mission? 


Not 

Important 


Somewhat 

Important 


important  imporU 


M&S  Importance  ® 


© 


© 


© 


7.  What  best  describes  the  primary  use  of  M&S  for  your  organization? 
©  Mission  Planning  and  Assessment 
©Training 
©  Experimentation 
• .  Research  or  Tradespace  Analysis 

©  Other,  please  specify _ 


S.  What  models,  simulations,  and  associated  tools  does  your  current  organization  use? 


9.  What  are  the  biggest  limitations  of  the  models,  simulations,  and  associated  tools  you  currently  use? 
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10.  What  is  the  typical  duration  of  Modeling  and  Simulation  phases  for  an  M&S  event? 


<2  days 

2  Days  -  2 

Weeks 

2-6  Weeks 

>6  Weeks 

Systems 

Engineering 

© 

© 

© 

Development 

© 

rO" 

© 

© 

Data  Engineering 

© 

© 

© 

© 

Testing 

© 

© 

© 

Execution 

© 

© 

© 

*4 

Analysis 

© 

© 

a 

<a 

11.  Has  your  organization  ever  used  a  distributed  simulation  approach? 

©Yes 

m  Mo 

12.  How  frequently  does  your  organization  use  DISTRIBUTED  simulation? 

Daily 

Weekly 

Monthly 

A  Few 
times  per 
Year 

Never 

Frequency  of 
Distributed 

Simulation  Use 

m 

© 

© 

© 

© 

13.  What  is  the  typical  Classification  level  at  which  your  organization  conducts  M&S  work? 

Unclassified 

FOUO 

Secret 

Above 

Secret 

Level 

Classification 

(§> 

© 

© 

© 

14.  How  Important  is  Verification,  Validation,  &  Accreditation  with  respect  to  your  organization's  use  of  M&S7 


Not 

Important 


Somewhat 

Important 


Important 


Show 

Stopper 


VV&A  Importance 


© 


>4 


© 


IS.  What  percentage  of  the  models,  simulations,  and  associated  tools  you  currently  use  are  Verified,  Validated,  & 
Accredited? 

'  0-25  Percent  25-50  Percent  '<*  50-75  Percent  _  75-100  Percent 


16.  From  your  perspective,  rank  order  the  importance  of  the  following  criteria  with  respect  to  a  combat  simulation 
package  (1  being  least  important,7  being  most  important): 


1 

2 

3 

4 

5 

6 

7 

Low  barrier  to  use  for 

varying  M&S  skill 
levels 

®> 

© 

© 

© 

© 

© 

© 

Ability  to  Access  and 

Run  from  anywhere 

© 

© 

© 

© 

© 

© 

Integrates  with  other 
commonly  used  M&5 
Packages  or  Scripts 

© 

© 

© 

© 

© 

© 

Powerful  and 
customizable  output 

© 

0 

© 

© 

© 

© 

graphics  and  statistics 
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Front  end  DOE 

capability  to  plan  and  ©  # 

customize  experiments 

©  © 

© 

Ability  to  add  and  ^ 

modify  scenarios 

©  © 

© 

Ability  to  interface  with 

and  draw  from  ^  «. 

authoritative  data 

©  © 

© 

sources 

17.  From  your  organization's  perspective,  select  the  TWO  most  important  considerations  in  selecting 

a  most 

appropriate  combat  simulation  program. 

Please  select  2  choices 

0Low  barrier  to  use  for  varying  M&S  skill  levels 

0  Ability  to  Access  and  Run  from  anywhere 

0 Integrates  with  other  commonly  used  M&S  Packages  or  Scripts 

0 Powerful  and  custom izeable  output  graphics  and  statistics 

0 Front  end  DOE  capability  to  plan  and  customize  experiments 

0 Ability  to  add  and  modify  scenarios 

0 Ability  to  interface  with  and  draw  from  authoritative  data  sources 

18.  What  M&S  PRE-processing  tools  do  you  commonly  use? 

19.  What  M&S  POST- processing  tools  do  you  commonly  use? 

20.  How  many  simulation  engineers  are  involved  in  your  typical  simulation  event? 

1  2-5 

6-19 

20+ 

Number  of  t  -  0 

Engineers 

© 

© 

21.  How  often  are  your  models,  simulations,  and  associated  tools  CHANGED  (including  data,  configuration,  design,  or 

algorithms)? 

A  Few 

Once  or 

Daily  Weekly  Monthly 

times  per 

Twice  a 

Year 

Year 

Simulation  ^  a 

Changes  ° 

© 

© 

22.  How  often  do  you  develop  NEW  SCENARIOS  for  your  simulations? 

A  Few 

Once  or 

Daily  Weekly  Monthly 

times  per 

Twice  a 

Year 

Year 

New  Scenario 

Development 

© 

© 

23.  How  many  people  are  typically  involved  in  creating  new  scenarios? 

1  2-3 

4-5 

More  than 

5 
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Scenario 

Development 


# 


© 


© 


© 


24.  What  standard/  form  at  is  used  to  digitally  save  your  scenarios  (include  version  if  known)? 


25.  Are  humans  required  to  interact  with  your  typical  simulation  during  its  run  for  pucking,  monitoring,  etc? 
©Yes 
#  No 


26.  What  percentage  of  your  organization's  modeling  and  simulation  tools  require  human  in  the  loop  interaction? 
©0  ®  <25%  0  25-75%  ©75  -  99%  ©100% 


27.  How  much  time  is  does  it  typically  take  to  INITIALIZE  a  simulation  once  it  has  been  developed  for  use  (assumes 
data  already  loaded)? 


Simulation 

Initialization 


<10  Minutes 


10-30 

Minutes 


30-60  1  or  More 

Minutes  Hours 


© 


© 


© 


28.  How  much  time  is  does  it  typically  take  to  CONFIGURE  a  simulation  once  it  has  been  developed  for  use  (assumes 


data  already  loaded)? 

<10  Minutes 

10-30 

30-60 

1  or  More 

Minutes 

Minutes 

Hours 

Simulation 

Configuration 

© 

© 

© 

29.  How  are  your  Simulations  executed? 
'  Single  Machine 

Local  Area  Network  (LAN) 

■  ‘  Wide  Area  Network  (WAN) 

Other,  please  specify 


Back 


Next 


Cancel 
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EASE 

Page  6  of  7 

EASE  Feedback 

30.  How  would  EASE  meet  some  of  your  organization's  M&S  needs? 


31.  From  your  perspective,  incorporation  of  what  features  or  functionality  into  EASE  would  increase  your  likelihood  of 
use? 


32.  What  specific  M&S  representations  (i.e.  CBRN,  Cyber,  etc.)  would  your  organization  like  to  see  incorporated  into 
EASE? 


33.  Are  you  aware  of  any  other  organization  or  agency  working  on  an  effort  similar  to  EASE? 

©Yes  ©No 

O  If  Yes,  Who  or  What  Project? 

34.  EASE  can  incorporate  the  use  of  surrogates  in  a  simulation.  Surrogates  are  plug  and  play  modules  that  replicate 
essential  model  components  that  are  currently  not  available.  How  could  you  use  the  surrogate  functionality? 


35.  Would  you  need  a  new  Certificate  of  Networthiness  to  run  something  like  EASE? 
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EASE 

Page  7  of  7 

Last  chance... 


37.  If  you  would  like  to  make  any  comments  on  the  topics  of  this  survey  or  any  other  M&S  topic  of  interest  to  you 
and/or  your  organization  that  were  not  addressed  in  this  survey,  please  type  them  in  the  space  below. 


For  further  information  about  this  project: 

Gene  Lesinski 

Operations  Research  Center 
Department  of  Systems  Engineering 
United  States  Military  Academy 
ATTN:  ORCEN 

Building  752  Mahan  Hall  Room  4th  Floor 
West  Point,  NY  10996 

TELEPHONE:  Commercial  (845)  938-5897  DSN  688-5897  EMAIL:  Eugene.Lesinski@usma.edu 


Back  |  [  Done  |  Cancel  | 
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Appendix  C.  EASE  Survey  Results 


Years  of  Combat  Simulation 


Role  Related  to  Combat 


Experience 


Simulation 


60 
50 
S,  40 

<u 

I  30 

u 

£  20 
10 
0 
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1 

18  19 

III 

1-3  3-5  5-8  >8 

Years 


Frequency  of  Simulation  Use 


Importance  of  Simulation 


Daily  Weekly  Monthly  Few  Once  or 
Times  Twice  a 
Per  year  Year 


Frequency 


Not  Somewhat  Important  Very 
Important  Important  important 

Importance 
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Duration  of  Systems  Engineering 
Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 

Duration 


Duration  of  Development  Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 


Duration 


Duration  of  Data  Engineering 
Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 

Duration 


Duration  of  Testing  Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 

Duration 


Duration  of  Execution  Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 

Duration 


Duration  of  Analysis  Phase 


<2  days  2  days-2  2-6  Weeks  >6  Weeks 
Weeks 

Duration 
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Yes  No 


Frequency  of  Federated 
Simulations 


Daily  Weekly  Monthly  A  Few  Never 
Times  a 
Year 


frequency 


Classification 


VV&A  Importance 


60 

50 


UN  CLASS  FOLIO 


SECRET  >SECRET 


Not  Somewhat  Important  Show 
Important  Important  Stopper 

Importance 


Models  and  Tools  That  are 
VV&Ad 


0-25  25-50  50-75  75-100 

Percentage  of  Tools 
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Low  Barrier  to  Use  Importance 


Level  of  Importance  (1  =  Low,  7  =  High) 


Ability  to  Access  from  Anywhere 
Importance 


Level  of  importance  (1  =  Low,  7  =  High) 


Importance  of  Integrating  with 
other  M&S  Models  or  Tools 


1  2  3  4  5  6  7 

Level  of  Importance  (1-Low,  7=High) 


Importance  of  Custom izeable 
Output 


1  2  3  4  5  6  7 

Level  of  Importance  fl=Low,  7=High) 


Importance  of  Front  End  DOE 
Capability 


12  3  4  5  6  7 


Level  of  Importance  (l=Low,  7=High) 


Add  and  Modify  Scenario 
Importance 


40 

35 

30 

,25 

20 

15 

10 

5 

0 


34.78 


15.22 


6.52  6.52 


4.35 


-6^5- 


_  B _ B _ b_  B 


■^26t69" 


1  2  3  4  5  6  7 

Level  of  Importance  (l=Low,  7=High) 
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Importance  of  integrating  and  Drawing  from 
Authoritative  Data  Sources 


1  2  3  4  5  6  7 


Level  of  Importance  {l=Low,  7= H igh) 


Number  of  Engineers  Required 


63703 


1  2-5  6-19  20+ 

Engineers  Required 


Frequency  of  Simulation  Changes 


Daily  Weekly  Monthly  Few  Once  or 
Times  a  Twice 
year 


Frequency 


Frequency  of  New  Scenario 
Development 


Daily  Weekly  Monthly  Few  Once  or 
Times  a  Twice 
year 


Frequency 


Number  of  Scenario  Developers 


1  2-3  4-5  >5 

Number  of  Developers 
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Simulation  Initialization  Duration 


60  — 5532 


<10  10-30  30-60  1  or  More 

Minutes  Minutes  Minutes  Hours 

Duration 


Simulation  Configuration 
Duration 


<10  10-30  30-60  lor  More 

Minutes  Minutes  Minutes  Hours 

Duration 


Simulations  Executed  On 


80 


Single  LAN  WAN  Other 
machine 


Type  System 


Aware  of  Others  With  EASE-Like 
Product 

120 
100 


c  60 

<u 

y 

£  40 

20 
0 

Yes  No 


97 

3 

Require  a  CON  to  Operate  EASE 


Yes  No 
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Sims,  Mode  Is,  Tools 

Freq 

JCATS 

23 

VBS2 

19 

OneSAF 

12 

Combat  XXI 

9 

ERF 

8 

FIRESIM 

7 

JDLM 

7 

LOD 

6 

A  WARS 

6 

CPOF 

5 

TIGR 

4 

SIMPLE 

4 

MRF 

3 

UAS 

3 

JCR 

3 

WARSIM 

2 

IWARS 

1 

m  Toolkit 

1 

JIB  US 

1 

BCMS 

1 

HELIX 

1 

URBAIMSIM 

1 

JSAF 

1 

JIMM 

1 

Co  1ST 

1 

What  apears  to  be  useful  about:  EASE 

Easy  to  configure  Federates  that  are  Correlated _ 

Low  overhead  driverto  stimulate  Army  mission  Command  Systems 
Interact  with  Army  Mission  Command  Systems _ 

Increase  Output  data  from  other  M&S _ 

Identify  outdated  simulations  which  could  be  replaced 
Interoperability  between  M&S  systems _ 

Use  as  a  screeningtool  for  more  high  resolution  tools 


Desired  Additional  Capabilities 

CESRN 

4 

Cyber 

3 

C2 

2 

Intel 

2 

Direct  and  indirect  fire 

2 

lEDs 

1 

Sensors 

1 

UAS 

1 

Non- Lethal 

1 

What  Would  Increase  Chances  of  Using  EASE 
Easy  to  use  information  exchange  and  data  mapping 
Interoperability  with  Army  MCS 
Ability  to  customize/change  scenarios _ 

Ability  to  integrate  with  other  M&S 
Provide  AAR  info 
Produces  accurate  data 

Ease  of  use _ 

Auto  configuration  and  launching 

Native  support  for  TENA _ 

Adjust  rapidly  to  new  code,  behaviors,  and  data 
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Pre-processing 

Order  Of  Battle  Serves  (OBS)JLCCTC  Validational  Tools 

Joint  Training  Data  Services 

Post- processing 

JTDS,  Joint  Remote  Client 

After  Action  Review  System  (AARS}  that  is  part  of  JLCCTC  ERF 

APE:  AWARS  Pre-processing  Environment 

AWARS  Post-processor,  SAS 

IPR,  MS  EL  Sync  meeting,  datbase  creation,  RTOC  setup 

AAR'S,  ICE  FORMS 

LOD 

LOD 

Polaris 

GUI  standard  output  packages 

A  MS  A  A  Joint  Data  Center  (JDC) 

NSITE 

SAF 

In-house-developed  data  reduction  software 

SIMPLE 

Microsoft  Excel,  Microsoft  Word,  Microsoft  SQL  Server 

COMBATXXI  Preprocessor  (AKA  Scenario  Integration  Tool  Suite) 

Custom  SQL  Server-based  post  processor.  Excel. 

AWARS  uses  two  preprocessors:  Ape  and  FSMP. 

SAF/AAR 

SQL,  Excel 

PASW  (AKA  Clementine),  Excel,  and  SQL. 

Electronic  Data  Request  System  (eDRS) 

SQL  Data  base  and  Excel. 

ArcGIS,  TerraSim  TerraTools 

Open  Office,  data  tools  include  MS  Access  and  MS  Excel 

_ Scenario  Formats _ 

Save  scenarios  in  each  federate  (VB32,  QneSAF,  etc...)' 

QBS  Y2.0 _ 

MS  Powerpoint/Word,  Adobe  Reader,  C2PC  (Command  and  Control  Personal  Computer] 
CD _ 

Disks,  Sharepoimt, 

VBS2  Format _ 

QneSAF  -  XML  (BZIP'd) _ 

SAF _ 

JCAT5VBS2 _ 

RTF  files  or  xml _ 

XML _ 

XML  and  odb  (open  office  database). 

Word  and  PowerPoint. 


51 


Appendix  D.  Workshop  Biggest  M&S  Shortfalls 


Organization 

Mission 

Biggest  M&S  Shortfalls 

Ft.  Hood  Mission  Command 

Training  Center 

Support  unit  training  objectives 
using  the  Army's  Mission 

Command  Systems 

Maintaining  relevancy  with  the  training  unit  requirements.  Changes  in 
MCS.  Scenarios  (terrain,  equipment). 

Ft.  Benning  Maneuver 

Battle  Lab 

Recommend  DOTMLPF  solutions 
based  on  LVCAR&G  experiments 

Inability  to  provide  an  integrated,  comprehensive  Common  Operating 
Picture  (COP)  to  all  Joint  Mission  Command  Systems  in  the  current 
Simulation  Architecture.  Applies  to  Air  and  Missile  Defense 
Workstation  (AMDWSj,  Forward  Area  Air  Defense  -  Engagement 

USMA  Department  of 
Systems  Engineering 

Educate  West  Point  cadets 

Technical  knowledge  and  talent  availability  /  reliance  on  small  staff. 

Ft.  Stewart  Mission  Training 
Complex 

Training  and  facility  support  to 

units  in  LVC simulation  exercises 

at  all  levels 

M&S  Terrain  availability  to  match  terrain  used  in  Mission  Command 
systems.  Interoperability  across  all  M&S  systems. 

Ft.  Leavenworth  TRADOC 

Analysis  Center  (TRAC) 

Operations  analysis  to  inform 
decisions  across  the  spectral 
(coneptsto  operations) 

Mission  Command  and  Cultural  Effects 

Army  Capabilities 

Integration  Center  (ARCIC) 

Development  and  integration  of 
force  capabilities  across  the 

DOTMLPF 

Representing  all  current  and  future  Warfighting  Functions  accurately 
enough  (entity  level)  while  addressing  higher  echelon  concepts  (Div 
&  JTF) 

Test  and  Training  Enabling 
Architecture 

Architecture  and  software 
necessary  to  enable  testing  and 
training  on  ranges 

Lack  of  full  suite  of  object  models.  DDM  support. 

Ft.  Campbell  Mission 
Training  Complex 

Mission  Command  digital  training 

for  multi-echelon  combined  arms 

operations 

Scenario  generation  timelines.  Formal  system  training.  Funding 
challenges  with  respect  to  class  availability  such  as  terrain  building 
and  scripting.  Integration  between  current  simulations 

Army  Materiel  Systems 
Analysis  Activity  (AMSAA) 

Conduct  analyses  across  Materiel 
Life  Cycle  informing  Army 
decisions 

Maintaining  supporting  infrastructure  for  a  broad  array  of  specific 
systems'  operating  requirements,  data  storage  and  retrieval,  and 
search  engine  to  enable  study  development  and  M&S  tool  upgrades 

USAOTCTest  Technology 
Directorate  (TTD) 

Support  for  test  directorates  in 
execution  of  operational  test  in 
joint  environments 

Nonfunctional  analysis  (e.g.,  performance,  scalability)  of  M&S  system 
of  systems  in  relation  to  system  under  test. 
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Appendix  E.  EASE  Workshop  Feedback  Comments 


EASE  Positives: 

Capture  interoperability  requirements  between  simulations/tools 

Captures  technical  knowledge 

Maintains  repository  of  M&S  (local  and  central) 

Simplify  the  process  of  configuring  and  running  applications  like  Combat  XXI  (currently  takes  several 
hundred  lines  of  code) 

Administrative  side:  Central  management  tool  to  launch  multiple  apps  from  a  central  point  -  will  help 
administrators'  jobs  like  turning  on  boxes  managing  OS  updates,  configurations,  etc. 

Stair-step  approach  at  training  facility  to  test/retest  not  possible  because  tech  staff  gone  so  having  EASE 
capture  test  results  for  rerunning  the  test  (regression  testing)  -  repeatability  to  tests 
-  testing  errors  based  on  modifications 

Butler:  See  potential  but  still  need  to  look  at  a  lot  of  things  for  ARCIC  environment 

Dison:  If  it  could  identify  capability  gaps  within  current  model,  find  other  models  in  environment,  bring 

other  application  in  to  fill  gap.  Zinser:  Only  applicable  if  system  at  DoD  level. 

Saluto:  At  DoD  level,  it  could  find  redundancies  to  and  remove  them. 

Butler:  How  does  EASE  interview  know  which  model  is  correct  per  use.  (How  to  distinguish  between 
OneSAF  and  J  SAF  -  both  entity  models,  but  what  are  the  important  differences  to  show  to  the  user) 

Bayer:  Need  filtering  to  distinguish  between  capabilities.  Should  allow  for  political  drivers  -  "You  will  use 
OneSAF" 

Dison:  With  every  study  requirement,  they  do  workshops  with  users  "measurement  space  workshop"  to 
work  out  issues  they're  trying  to  analyze,  alternatives  of  models,  scenarios,  etc.  Be  nice  if  EASE  could 
help  with  the  measurement  space  paring  down.  EASE  could  help  with  the  process  -  right  now  the  process 
is  BOGSAT  and  Microsoft  Office  products,  (low  priority) 

Wood:  Building  simulations  around  capabilities  rather  than  applications  is  a  positive.  Also,  the  architecture 
capture  aspect  of  the  SDD 

Excitement  about  virtual  machine  usage/management. 

EASE  Negatives: 

Bayer:  Single  point  of  failure  if  something  happens  to  the  server.  Answer:  Automatic  backups  - 
distributed  configuration  repository.. .could  be  off-site  backup.  Enterprise  investment  in  the  technology 
through  the  rapid  changes  in  the  industry.  DISA  /  DIACAP  /  CON  /  ATOs  approval  will  drive  this  in  future. 
Bayer:  Not  certain  that  entity  mapping  will  translate  in  the  interoperability.  Platform  enumerations 
translation  between  systems  verification. 

Butler:  Terrain  correlation  between  simulations. 

Bayer:  What  about  gateway  mappings  making  this  more  cumbersome? 

Saluto:  Time  and  effort  to  initially  get  currently  used  systems/tools  into  the  left-side  of  EASE  (SDD,  test 
cases,  etc.) 

Zinser:  Nobody  except  for  the  systems  developer  could  fill  in  the  SDD  details.  Both  knowledge  and 
time/capacity. 

Bolton:  They  get  systems  as  black  boxes  (even  as  disk  images  instead  of  DVDs)  that  they  don't  really 
understand  the  details  of  how  they  work  -  can't  start  putting  details  into  SDD. 

Zinser:  What  if  an  Army  regulation  for  M&S  developers  were  to  provide  all  the  systems  engineering 
details  in  their  development  schedule  -  they're  required  to  get  the  details  in  a  format  that  EASE  could 
take  advantage  of 

Saluto:  Linked  VBS2  and  LaserShot  -  difficult  even  with  developers  available,  but  this  is  a  one-off. 

Tough  to  find  the  right  middle  ground  between  getting  too  much  detail  and  too  little  detail 
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Bolton:  Already  have  working  system,  don't  need  EASE  to  help.  It  is  what  it  is  and  they  have  the 
necessary  support  already 

Bayer:  Current  EASE  deployment  mechanism  won't  support  deploying  to  specific  IP  addresses  -  current 
hardware  setup  (show-stopper  so  should  be  high  priority) 

Saluto:  Configuration  management  help?  Answer:  EASE  certainly  helps  with  capturing  systems  as  parts 
rather  than  CD  black  boxes 

Dison:  Aside  from  up-front  work,  trust  that  this  will  be  the  long-term  solution  because  TRAC  is  a  one- 
stop-shop  so  to  relinquish  control  to  EASE  and  then  something  happens  like  breaking,  lack  of 
funding/support,  etc.  then  the  negative  would  be  a  severe  impact.  Would  be  nice  if  DoD-backed  / 
mandated  projects,  (organizational  risk  of  drastically  changing  process  and  technical  solutions). 

Saluto:  If  I  were  PM  ConSim,  push  for  business  practices  -  implement  ourselves  for  others  to  take 
advantage  of  -  since  PM  ConSim  institutionalizing  EASE,  confidence  is  higher  that  the  system  1)  works 
and  2)  will  be  around  for  awhile. 

Bayer:  A  lot  of  free-form  in  the  SDD  (english  text)  and  not  strict.  Software  system  should  have  a 
requirement  to  meet  a  certain  depth  requirement  of  data  -  should  have  controlling  body  /  accountability  / 
quality  control  for  SDD  data  entry. 

Need  to  address  multiple  object  models  in  the  SDD  (same  version) 

EASE  I  mprovement  /  Extension  I  deas: 

Zinser:  Parametric  data  linking  -  fair  fight  issues  -  systems  engineering  tool  to  determine  Ph/PK  table 
good  enough  (medium) 

Zinser:  Enumeration  comparison  /  mapping  capability  (high  priority)  [JG:  We  could  help  with  the  mapping 
output  file  a  la  pub/sub  matrix  for  visual  comparison] 

Bolton:  Could  sell  EASE  better  if  it  could  leverage  network  in  place  compared  to  hub-spoke  concept.  How 
to  manage  business  model  /  politics  of  linking  across  larger  organization  (low) 

Carr:  Information  not  very  good  coming  out  of  OneSAF  artifacts.  How  could  EASE  allow  for  better 
definition  of  output  metrics  /  analysis  aids  (what  data  /  views  to  grab)  -  customizable  output  for  study, 
(low) 

Sipp:  Tie  in  global  URN  /  task  organization  (force  builder)  for  entity  building  for  easy  scenario.  Specific 
icons  on  display,  not  generic  tanks,  (medium  priority) 

Saluto:  Chris  Black  under  SI  MCI  -  UT.  C3T  tried  to  do  something  with  ASIS  products  to  have  LDIF  data 
into  simulation  (JCATS)  -  correct  LDIF  to  show  up  on  the  COP  display.  Not  an  EASE  target.. .database 
target  that  many  have  tried  to  resolve  -  so  far  not  there. 

Bayer:  Should  be  aligned  with  scenario  development  products.  Force  builder  to  generate  LDIF  specific  to 
an  organization  (Paul  Monday  willing  to  look  into  this)  (medium) 

Dison:  Use  case  would  be  for  a  study  /  scenario  using  large  scale  model  like  Combat  XXI,  zooming  into 
smaller  area  for  urban  environment  -  dynamically  change  model  representations  for  dynamic  resolution 
changes  via  switching  models.  Other  models  integrated  in  to  look  at  cultural  affects,  etc.... results  into 
aligned  data  afterwards,  (high  priority) 

Saluto:  Tried  by  several  -  but  never  knocked  true  interoperability  issues 

Link  to  WebMSDE  (medium  priority  because  nobody  uses  MSDE  even  in  future  they'll  use  different  tools) 
SDDs  for  common  tool  -  pick  some  common  items  like  JCATS,  RPR,  etc.  maybe  by  picking  a  domain  and 
starting  there,  (high  priority) 

Bayer:  Combine  getting  other  apps  and  representing  their  details  in  the  SDD 

Saluto:  Do  SDD  for  OneSAF  and  WarSim  and  go  show  ConSim.  Other  organizations  and  their  respective 
models  also.  Show  other  organizations  value  from  previously  done  domains.  Would  need  to  do  this  as  a 
federation  and/or  view  something  like  OneSAF  as  a  tool  -  SDD  adjustments.  Need  to  get  high-level  buy- 
in,  not  in  a  lab  without  much  influence  on  the  community.  This  needs  to  be  considered  when  STTC 
decides  where  to  deploy  an  initial  case  of  EASE. 
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Appendix  F.  Manager  Interview  Questions 

Before  Describing  EASE  Stakeholder  Questions 

•  Organization 

-  What  is  your  organization’s  mission? 

-  How  does  M&S  support  the  mission? 

-  How  frequently  do  you  use  M&S? 

-  How  long  is  each  M&S  event  including  the  entire  lifecycle?  (1  day,  week,  month, 
etc.) 

-  What  are  the  biggest  limitations  of  the  tools  that  you  currently  use? 

•  Execution 

-  Have  you  ever  used  a  federated  simulation  and  if  so,  how  often? 

-  If  you  were  going  to  compare  two  simulation  models  what  criteria  would  you 
use? 

-  What  are  your  V&V  requirements? 

-  What  classification  level(s)  do  you  execute  at? 


After  Describing  EASE  Stakeholder  Questions 

•  How  could  you  use  a  tool  like  this? 

•  What  is  missing  and  required  for  you  to  use  EASE? 

•  What  interface  would  be  required  to  facilitate  your  use  of  simulation? 

•  Are  you  aware  of  anyone  doing  something  similar? 

•  What  functionality  sounds  most  useful? 

•  Would  you  use  the  surrogate  functionality 

•  Will  you  need  a  new  CON  to  use  and  how  hard  will  it  be  to  get  one? 

•  Once  developed,  who  should  own  it? 

•  Technical  (as  applicable  and  interest  high  enough) 

-  Describe  the  life  cycle  of  an  M&S  execution.  How  long  does  each  phase  typically 
last?  (i.e.  4  weeks  systems  engineering,  12  weeks  development,  10  weeks  data 
engineering,  2  weeks  testing,  1  week  execution) 

-  How  many  engineers  are  involved? 

-  How  often  are  models  changed  (including  data,  configuration,  design  or 
algorithms)? 

-  Describe  your  pre-processing  and  post-processing  tools  and  processes 

-  How  are  scenarios  developed  and  captured/represented? 

-  Describe  the  system  initialization  /  startup  process 

-  Are  humans  required  to  interact  with  the  M&S  for  training,  pucking,  monitoring, 
etc.? 

-  Is  specialized  hardware-in-the-loop  required? 

-  For  distributed  simulation,  what  protocols  are  used? 

-  Do  you  execute  locally  or  over  a  Wide  Area  Network? 
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Appendix  G  Findings,  Conclusions,  and  Recommendations 


Scenarios 


Finding 

Conclusion 

Recommendation 

Ability  to  modify  or  develop  a  new  scenario 
identified  as  the  most  important  characteristic  by 
survey  respondents  and  workshop  participants 

Scenarios  are  created  and  saved  in  a  variety  of 
formats  to  include:  OBS  V2.0T  MS, 

Powerpoint/Word,  Adobe  Reader,  C2PC  [Command 
and  Control  Personal  Computer),  OneSAF  -  XML 
(BZIP'd),  and  others 

Need  exists  to  increase  speed  and  ease  of 
scenario  changes  and  development  in  a 
standard  format 

EASE  enhancements  include  rapid  scenario 
modification  and  development  capability 

Scenarios  are  typically  changed  monthly  ora  few 
times  peryear 

Typically  takes  2-3  scenario  developers  and  in  some 
cases  more  than  5 

Scenario  modifications  are  currently  time 
and  resource  intensive 

25%  of  organizations  noted  that  they  require  more 
than  5  scenario  developers  to  generate  a  new 

scenario. 

Typically  takes  several  weeks  to  develop  a  new 

scenario 

Data 


Findings 

Conclusions 

Re  co  m  m  e  ndatio  ns 

Requested  data  often  takes  months  to  be 

delivered. 

A  lot  of  time  and  resources  are  devoted  to 
validating  input  data 

Authoritative  input  data  is  difficult  and 
time  consuming  to  obtain 

EASE  enhancements  include  the  capability 
to  integrate  and  draw  from  authoritative 

data  sources 

The  ability  to  integrate  and  draw  from 
authoritative  data  sources  was  highlighted  as  a 
top  3  characteristic 

There  are  often  issues  of  data  compatability 
between  models  and  tools 

There  is  a  diverse  array  of  in-house  scripts  and 
tools  used  to  process  data  priorto  and  after 
simulation  execution 

Battle  labs  report  a  lack  of  data  and  scenarios 
that  highlight  differences  between  alternatives. 

Terrain  data  is  not  available  for  particular 
geographic  regions  for  all  models 

Terrain  data  availability  and  compatability 

are  an  issue 

Increase  the  EASE  terrain  availability  in  the 
line-up 

Terrain  data  is  does  not  correlate  or  is  not 
compatable  from  model  to  model 
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Hardware  &  Software  Footprint 


Findings 

Conclusions 

Recommendations 

Each  combat  simulation  lab  or  analysis  center 
maintains  computers,  servers,  and  specialized 
equipment  to  support  M&S  for  experimentation, 
training,  and  analysis. 

76%  of  survey  respondents  stated  that  they  execute  on 
a  single  machine  or  LAN  with  67%  operating  on  the 

LAN 

FT  Stewart  MTC  maintains  a  53K  sq  ft  f  acility  and  35 

servers. 

There  is  tremendous  redundancy  in  hardware 
and  software  across  the  M&S  Enterprise 

A  cloud  -based  solution  with  virtual 
machines  has  the  potential  to  provide 
substantial  cost  savings  across  the  M&S 
Enterprise. 

The  maneuver  battle  lab  at  FT  Benning  maintains  a 
120,000  Sq .  Ft.  Co nstructi ve/V  i  rtua  l/Ga  m  i  ng  S  i  m  u  latio  n 
Facility  with  450  desktop  computers  and  50  servers. 

ARCIC  has  no  standard  footprint  but  their  largest 
exercise  requires  300+  computing  platforms  (Sim,  MC, 
infrastructure,  etc) 

Similar  hardware  and  software  is  replicated  at  every 
lab,  MTC,  or  analysis  center 

VV&A 


Findings 

Conclusions 

Recommendations 

Managers  expressed  concern  with 
adopting  EASE  due  to  its  lack  of  VV&A 
certification 

The  specific  requirements  for  VV&A  of  a 
combat  simulation  model  are  general  and 
somewhat  vague 

VV&A  is  an  important  consideration  and  a 
potential  roadblock  to  transition  and  PGR 

status 

Users  are  generally  less  concerned  with 
VV&A  requirements  than  managers 

Identify  an  appropriate  VV&A 
authority  and  discuss  the  specific 
VV&A  requirements  as  they 

87%  of  survey  respondents  stated  that 

V  V&A  was  very  important  within  their 
organizations 

Most  report  that  over  75%  of  the  models 
and  tools  they  use  on  a  day-to  day  basis 
are  VV&Ad. 

Begin  action  on  those  VV&A 

related  tasks  that  can  be 
completed  now. 

The  TEMO  domain  VV&A  is  less 

concerned  with  VV&A  than  ACR  and  RDA 

domains. 

Emphasis  regarding  VV&A  varies  amongst 
domains 
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Inteoperability 


Findings 

Conclusions 

Re  co  m  mend  ati  o  ns 

Numerous  combat  functions  are 
replicated  between  combat 
simulation  models  and  tools 

There  are  redundancy  in  functions 
replicated  in  models 

With  the  fielding  of  new  equipment 
and  new  threats  additional  combat 

simulation  models  and  tools  are 
required 

The  application  line-up  database  and 
function  vs.  application  focus  are 
unique  and  positive  aspects  of  EASE 
and  should  be  leveraged 

Combat  simulation  models  and  tools 
were  not  necessarily  intended  to 
work  together  however  there  is  a 
requirement  to  integrate  multiple 
models 

Federation  of  multiple  combat 
simulation  models  and  tools  is 
increasingly  important 

Ability  to  integrate  multiple  models 
and  tools  was  classified  as  very 
important  and  listed  as  a  top  three 
characteristic 

Integrating  multiple  models  or  tools  is 
time  and  resource  intensive  and 
requires  specialized  experience. 

Federation  of  multiple  combat 
simulation  models  and  tools  is 
difficult  and  requires  specialized 
expertise 

EASE  extensions  should  focus  an 
increasing  the  timliness  and  reducing 
manpower  required  to  integrate 
multiple  models 

Reuse  and  Version  Control 


Findings 

Conclusions 

Recommendations 

Stakeholders  note  that  there  are  continuing 
challenges  with  configuration  due  to  changing 
versions  of  M&S  and  supporting  software 

Each  combat  simulation  lab  or  analysis  center 
maintains  an  extensive  staff  to  update  software 

Sotware  updates  and  version  control  are  not 
systematic  and  are  problematic 

Recommend  some  form  of  Reconfiguration 
savings  be  used  as  an  EASE  ROI  metric 

Stakeholders  identified  difficulty  with  keeping 
software  and  hardware  updated. 

Software  versions  are  developed  independently  of 
M&S  Enterprise  or  Program  integration  requirements 
and  generally  cause  issues  after  the  fact 

Updating  versions  of  combat  simulation 
models  and  supporting  software  is  resource 
and  time  intensive 

Highlight  EASE  ability  to  archive  previous 
working  models  with  appropriate 
configurations  and  versions 

Stakeholders  noted  great  value  in  the  ability  to 
access  previously  run  simulations  without  the 
burden  of  new  configuration  work  or  software 
updates. 

M&S  strategic  guidance  lists  reuse  as  an  important 
Enterprise  M&S  characteristic. 

Recommend  some  form  of  Reuse  be  used  as 

an  EASE  ROI  metric 

Stakeholders  state  that  the  ability  to  access  and 
rerun  any  simulation  archived  in  EASE  is  valuable  and 
potentially  a  tremendous  time  saver. 
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Manpower 


Findings 

Conclusions 

Recommendations 

Each  combat  simulation  lab  or  analysis 
center  maintains  an  extensive  staff  to 

maintain  hardware  and  software 

Survey  respondents  stated  that  they  spend 
a  tremendous  amount  of  time  maintaining 
hardware  and  software  to  support  M&S  in 

ther  labs 

There  is  tremendous  redundancy  in 
supporting  manpower  across  the  M&S 
Enterprise 

A  cloud  -based  solution  with  virtual  machines 

has  the  potential  to  provide  substantial  cost 
savings  across  the  M&S  Enterprise. 

FT  Hood,  FT  Stewart,  and  FT  Benning  MTC 

maintains  a  staff  of  between  35-70 

personnel  to  maintain  and  operate  M&S 

software  and  hardware 

TRAC-FLVN  has  a  staff  of  50+ personnel 
devoted  to  model  development, 
wargaming  and  analytic  tools 

Operating  and  Maintaining  M&S 
manpower  intensive 

Reuse  or  access  to  previous  M&S  information 
could  save  manpower  and  time 

For  major  events,  ARCIC  has  up  to  300 
personnel  devoted  to  technical  control, 
simulation  operation,  role  playing  and 
analysis 

Similar  personnel  resources  are  replicated 
at  every  lab,  MTC,  or  analysis  center 
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Appendix  H.  EASE  Metrics 


Term 

Definition 

Quality 

Monetary 

Impactful 

Inputs,  processes,  and 
outcomes  relative  to  the 
mission’s  goal  or  application 

use 

#  of  M&S  strategic  goals 
addressed 

#  of  M&S  capability  gaps 
addressed 

Organization  level  of  impact 
(i.e.  Enterprise,  Community, 
Program,  Lab) 

•  System  effects  the 
accomplishment  of  the 
mission  or  activity 

Cost  savings  when  impacts 
promote  efficiencies 

Cost  avoidance  when  impacts 
obviate  expenditures 

Innovative 

Includes  significant  new 
capabilities  or  provides 
functionality  in  an 
exceptional  way 

Duration  of  innovation  life 

#  of  innovation  concept 

reuses 

•  Analytic  functions  and 
implementation  are  unique 

Cost  savings  when 
innovations  reduce  labor, 
runtime,  etc. 

Cost  avoidance  through 
reduction  in  factors  not 

included 

Resource  Efficiencies 

Resources  needed 
(manpower,  equipment,  and 
software)  to  run  M&S 

#  of  servers,  computers, 
licenses  reduced 

#  of  personnel  reduced 

#  of  labor  hours  reduced 

•  System  requires  fewer 
resources  to  run/maintain 

Cost  savings  from  reduced 
equipment,  software 
purchases  and  upgrades 

Cost  avoidance  from  labor 

reduction 

Composability 

Can  be  quickly  reconfigured 
and  federated  with  others  via 

automated  tools 

#  of  additional  systems  that 
can  be  included 

#  time  required  to  include 
additional  systems 

•  System,  architecture, 
and  meta-data  allow 

automated  federation 

Cost  savings  from  combining 
systems  vice  new 

Cost  avoidance  from  reduced 
labor  to  interoperate  systems 

Interoperability 

Has  the  ability  to  be  modified 
in  a  timely  manner  to  pass/ 
receive  results/data,  syntactic, 
semantic  information 

6 

#  of  systems  it  can 
interoperate  with 

Degree  of  interoperability 

•  System  has  stable  and 

2  defined  interfaces  and 

Cost  savings  from  not  having 
to  develop  internal  modules 

Cost  avoidance  from  reduced 

labor  to  add  functions 

can  exchange  data 

Reuse 

Previous  models  can  be 
retrieved  and  rerun  yielding 
the  same  results  when  input 
conditions  are  the  same 

Time  required  to  access  and 
rerun  previous  model 

#  of  model  reuses 

•  System  allows  rerun  of 
previous  model  while 
preserving  previous 
configuration  and 
software  versioning 

Cost  savings  from  automated 
repetition 

Cost  avoidance  from  reduced 
labor  -  not  having  to  recreate 
and  reconfigure  a  previous 
model 

Adaptability 

Source  code  can  be  changed 
and  updated,  can  be  used  in  a 
different  application  area,  and 
can  be  altered  to  run  on  other 
systems/hardware 

#  of  components  and 
algorithms 

#  of  additional  applications 

Flexibility  of  input  files  and 
databases 

System  can  be  modified  to 
address  additional 
requirements  and  add 
functionality;  run  on  other 
systems 

Cost  savings  from  not  having 
to  develop  a  new  system 

Cost  avoidance  from  reduced 
time  to  update,  simplified  re¬ 
hosting,  and  labor  -reduction 
in  new  uses 

Transition 

Ownership  of  the  application 
is  successfully  transferred  to 
an  agency  outside  STTC 

Probability  of  Transition 

#  of  Months  to  Transition 

•  System  ownership  is 
successfully  transferred  to 
organization  outside 

STTC  and  requires  rare 
and  minor  assistance 

N/A 
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Nomenclature 


ACR 

AMRDEC 

AMSAA 

AMSO 

ARCIC 

ARL 

ATEC 

BLCSE 

BMC 

C2 

C2WT 

DEVS 

DIS 

DOD 

DODAF 

DSE 

EASE 

FACT 

FCR 

FOM 

HLA 

IWARS 

JCATS 

JSAF 

MATREX 

M&S 

MDA 

MSBL 


Advanced  Concepts  and  Requirements 

Aviation  and  Missile  Research  and  Development  Center 

Army  Material  Systems  Analysis  Agency 

Army  Modeling  and  Simulation  Office 

Army  Capabilities  and  Integration  Center 

Army  Research  Laboratory 

Army  Test  and  Evaluation  Command 

Battle  Laboratory  Collaborative  Simulation  Environment 

Brigade  Modernization  Command 

Command  and  Control 

C2  WindTunnel 

Discrete  Event  System  Specification 
Distributed  Interactive  Simulation 
Department  of  Defense 

Department  of  Defense  Architecture  Framework 

Department  of  Systems  Engineering 

Executable  Architecture  Systems  Engineering 

Framework  for  Assessing  Cost  and  Technology 

Findings,  Conclusions,  and  Recommendations 

Federation  Object  Model 

High  Level  Architecture 

Infantry  Warrior  Simulation 

Joint  Conflict  and  Tactical  Simulation 

Joint  Semi-Automated  Forces 

Modeling  Architecture  for  Technology  Research  and  Experimentations 
Modeling  and  Simulation 
Missile  Defense  Agency 
Maneuver  Support  Battle  Lab 
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MSCO 

MSCoE 

OneSAF 

ORCEN 

PaaS 

PEO  STRI 

RDA 

RDECOM 

RID 

RTI 

SDD 

SDP 

SOSI 

STTC 

SWOT 

TENA 

TEMO 

TOC 

TRAC 

TRADOC 

USMA 

VBS2 

VFT 

VV&A 


Modeling  and  Simulation  Coordination  Office 
Maneuver  Support  Center  of  Excellence 
One  Semi-automated  Forces 
Operations  Research  Center 
Platform  as  a  Service 

Program  Executive  Office  Simulation  Training  and  Research  Integration 

Research  Development  and  Acquisition 

Research  Development  and  Engineering  Command 

Requirements  Integration  Directorate 

Run  Time  Interface 

Software  Design  Description 

Systems  Decision  Process 

System  of  Systems  Interoperability 

Simulation  and  Training  Technology  Center 

Strengths,  Weaknesses,  opportunities,  and  Threats 

Test  and  Training  Enabling  Architecture 

Training,  Exercise,  and  Military  Operations 

Tactical  Operations  Command 

TRADOC  Analysis  Center 

Training  and  Doctrine  Command 

United  States  Military  Academy 

Virtual  Battle  Space  2 

Value  Focused  Thinking 

Verification,  Validation  and  Accreditation 
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